i upgrade my WHM server / Cent-OS 7 from MySQL 5.6 to 5.7 this server is dedicated server for whm server [apache, php, mysql] hosting for multi web sites with very small number of visitors. after that upgrade every few hours my server memory full by MySQL which after hours of MySQL restart it exceed 80% of whole memory. i increase my server ram from 13GB to 17.5GB and still get same problem also i create swap for 3GB and still same problem. i change in MySQL configuration without any luck. so if anyone can help with that.
i try also mysqltuner but the problem that MySQL restarting every few hours and they said that MySQL must be running more than 24 hour at lease for recommendation to be accurate.
this is the configuration of the etc/my.cnf file:
note i try to put innodb_buffer_pool_size to 6G as recommended but still the same memory load.
in mysql tuner result they gave result that Maximum possible memory usage: 57.2G (336.45% of installed RAM) is that a problem and how to fix this issue!
also no load on the server or mysql. and on top just the mysql is taking the memory above 70% of it
and this the output of free:
OOM Log Example:
Thanks,
i try also mysqltuner but the problem that MySQL restarting every few hours and they said that MySQL must be running more than 24 hour at lease for recommendation to be accurate.
this is the configuration of the etc/my.cnf file:
Code:
[mysqld]
default-storage-engine=INNODB
innodb_file_per_table=1
max_allowed_packet=268435456
open_files_limit=80000
collation_server=utf8_unicode_ci
character_set_server=utf8
slow_query_log=1
long_query_time=3
max_connections = 200
sort_buffer_size=16M
join_buffer_size=1M
innodb_sort_buffer_size=16M
read_buffer_size=1M
read_rnd_buffer_size=2M
max_heap_table_size=64M
tmp_table_size=64M
key_buffer_size=128M
#event_scheduler=on
innodb_log_file_size=384M
query_cache_type=0
query_cache_size=0
table_open_cache=78000
table_definition_cache=39400
innodb_buffer_pool_instances=3
innodb_buffer_pool_size=3G
and the result of mysql tuner i don't know if it can help as MySQL not running from a long time:
>> MySQLTuner 1.7.19 - Major Hayden <[email protected]>
>> Bug reports, feature requests, and downloads at http://mysqltuner.com/
>> Run with '--help' for additional options and output filtering
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.7.31-log
[OK] Operating on 64-bit architecture
-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/lib/mysql/bilal.hosting.err exists
[--] Log file: /var/lib/mysql/bilal.hosting.err(38K)
[OK] Log file /var/lib/mysql/bilal.hosting.err is readable.
[OK] Log file /var/lib/mysql/bilal.hosting.err is not empty
[OK] Log file /var/lib/mysql/bilal.hosting.err is smaller than 32 Mb
[!!] /var/lib/mysql/bilal.hosting.err contains 55 warning(s).
[!!] /var/lib/mysql/bilal.hosting.err contains 5 error(s).
[--] 5 start(s) detected in /var/lib/mysql/bilal.hosting.err
[--] 1) 2020-09-13T23:07:20.256464Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 2) 2020-09-13T23:07:15.490922Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 3) 2020-09-13T09:53:37.024396Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 4) 2020-09-12T11:38:30.571913Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 5) 2020-09-12T05:40:13.253650Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 1 shutdown(s) detected in /var/lib/mysql/bilal.hosting.err
[--] 1) 2020-09-13T23:07:17.257863Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA
[--] Data in MyISAM tables: 10.0M (Tables: 51)
[--] Data in InnoDB tables: 5.4G (Tables: 30538)
[OK] Total fragmented tables: 0
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.
-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] There is no basic password file list!
-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 7h 37m 6s (1M q [45.248 qps], 7K conn, TX: 2G, RX: 90M)
[--] Reads / Writes: 99% / 1%
[--] Binary logging is disabled
[--] Physical Memory : 17.0G
[--] Max MySQL memory : 57.2G
[--] Other process memory: 0B
[--] Total buffers: 3.2G global + 276.2M per thread (200 max threads)
[--] P_S Max memory usage: 72B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 5.4G (31.56% of installed RAM)
[!!] Maximum possible memory usage: 57.2G (336.45% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (74/1M)
[OK] Highest usage of available connections: 4% (8/200)
[OK] Aborted connections: 0.28% (22/7973)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[OK] Query cache is disabled by default due to mutex contention on multiprocessor machines.
[OK] Sorts requiring temporary tables: 6% (3K temp sorts / 45K sorts)
[!!] Joins performed without indexes: 3963
[OK] Temporary tables created on disk: 8% (136K on disk / 1M total)
[OK] Thread cache hit rate: 99% (8 created / 7K connections)
[!!] Table cache hit rate: 1% (39K open / 3M opened)
[!!] table_definition_cache(39400) is lower than number of tables(44053)
[OK] Open file limit used: 0% (386/80K)
[OK] Table locks acquired immediately: 100% (3K immediate / 3K locks)
-------- Performance schema ------------------------------------------------------------------------
[--] Memory used by P_S: 72B
[--] Sys schema is installed.
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.
-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.3% (24M used / 134M cache)
[OK] Key buffer size / total MyISAM indexes: 128.0M/1.1M
[OK] Read Key buffer hit rate: 99.8% (35K cached / 81 reads)
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[!!] InnoDB buffer pool / data size: 3.0G/5.4G
[OK] Ratio InnoDB log file size / InnoDB Buffer pool size: 384.0M * 2/3.0G should be equal to 25%
[OK] InnoDB buffer pool instances: 3
[--] Number of InnoDB Buffer Pool Chunk : 24 for 3 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 99.96% (414562897 hits/ 414708820 total)
[!!] InnoDB Write Log efficiency: 30.71% (1067 hits/ 3475 total)
[OK] InnoDB log waits: 0.00% (0 waits / 2408 writes)
-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is disabled.
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: ROW
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
Control warning line(s) into /var/lib/mysql/bilal.hosting.err file
Control error line(s) into /var/lib/mysql/bilal.hosting.err file
MySQL was started within the last 24 hours - recommendations may be inaccurate
Reduce your overall MySQL memory footprint for system stability
Dedicate this server to your database for highest performance.
Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
(specially the conclusions at the bottom of the page).
Increase table_open_cache gradually to avoid file descriptor limits
Read this before increasing table_open_cache over 64: xxxxxxxxxxxxxxxxxxxx
Read this before increasing for MariaDB https://mariadb.com/kb/en/library/optimizing-table_open_cache/
This is MyISAM only table_cache scalability problem, InnoDB not affected.
See more details here: https://bugs.mysql.com/bug.php?id=49177
This bug already fixed in MySQL 5.7.9 and newer MySQL versions.
Beware that open_files_limit (80000) variable
should be greater than table_open_cache (39895)
Variables to adjust:
*** MySQL's maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
join_buffer_size (> 1.0M, or always use indexes with JOINs)
table_open_cache (> 39895)
table_definition_cache(39400) > 44053 or -1 (autosizing if supported)
innodb_buffer_pool_size (>= 5.4G) if possible.in mysql tuner result they gave result that Maximum possible memory usage: 57.2G (336.45% of installed RAM) is that a problem and how to fix this issue!
also no load on the server or mysql. and on top just the mysql is taking the memory above 70% of it
and this the output of free:
Code:
total used free shared buff/cache available
Mem: 16G 13G 323M 14M 3.6G 3.6G
Swap: 3.0G 2.1G 933M
Total: 19G 15G 1.2G
Code:
# grep -i "out of memory" /var/log/messages*
/var/log/messages:Sep 7 03:00:23 hosting kernel: Out of memory: Kill process 1969 (mysqld) score 695 or sacrifice child
/var/log/messages:Sep 7 11:41:09 hosting kernel: Out of memory: Kill process 1787 (mysqld) score 756 or sacrifice child
/var/log/messages:Sep 7 13:25:46 hosting kernel: Out of memory: Kill process 8455 (mysqld) score 723 or sacrifice child
/var/log/messages:Sep 7 15:29:27 hosting kernel: Out of memory: Kill process 28063 (mysqld) score 734 or sacrifice child
/var/log/messages:Sep 7 20:56:49 hosting kernel: Out of memory: Kill process 23450 (mysqld) score 727 or sacrifice child
/var/log/messages:Sep 7 20:56:50 hosting kernel: Out of memory: Kill process 2043 (clamd) score 80 or sacrifice child
/var/log/messages:Sep 8 03:05:20 hosting kernel: Out of memory: Kill process 26713 (mysqld) score 681 or sacrifice child
/var/log/messages:Sep 8 05:28:19 hosting kernel: Out of memory: Kill process 3038 (mysqld) score 681 or sacrifice child
/var/log/messages:Sep 8 11:14:35 hosting kernel: Out of memory: Kill process 30391 (mysqld) score 742 or sacrifice child
/var/log/messages:Sep 8 15:02:51 hosting kernel: Out of memory: Kill process 3984 (mysqld) score 742 or sacrifice child
/var/log/messages:Sep 9 12:21:44 hosting kernel: Out of memory: Kill process 1743 (mysqld) score 763 or sacrifice child
/var/log/messages:Sep 9 14:31:21 hosting kernel: Out of memory: Kill process 16796 (mysqld) score 774 or sacrifice child
/var/log/messages:Sep 10 13:01:14 hosting kernel: Out of memory: Kill process 4654 (mysqld) score 788 or sacrifice child
/var/log/messages:Sep 10 13:01:14 hosting kernel: Out of memory: Kill process 2002 (clamd) score 61 or sacrifice child
/var/log/messages:Sep 10 16:31:45 hosting kernel: Out of memory: Kill process 12720 (mysqld) score 816 or sacrifice child
