MySQL Keeps Crashing

[gseven]

Hello everyone,
Following is the email that server sent:

Out of memory: The process “mysqld” was terminated because the system is low on memory.
In order to avoid a system crash due to low memory, the kernel terminated the process named “mysqld” with the PID “12653”.

Process Name mysqld
Event Time Saturday, March 25, 2017 at 6:47:07 AM UTC
PID 12653
Process UID 496
Process Username mysql
Process Total Virtual Memory 923976kB
Process Anonymous Resident Set Size 272236kB
Process File Resident Set Size 36kB
Process OOM Score 142
Status Out of Memory
Memory Information
Used 395 MB
Available 1.45 GB
Installed 1.83 GB
Load Information 4.27 1.59 0.62
Uptime 1 day, 19 hours, 50 minutes, and 22 seconds
IOStat Information avg-cpu: %user %nice %system %iowait %steal %idle 4.21 0.03 0.94 0.44 0.01 94.35 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 8.17 214.93 133.23 33920964 21025952
Top Processes
PID Owner CPU % Memory % Command
13814 root 6.58 1.78 tailwatchd - chkservd - sending OOM notification
13813 root 6.58 1.77 tailwatchd - chkservd - sending OOM notification
13811 root 6.49 1.77 tailwatchd - chkservd - sending OOM notification
13816 root 6.67 1.75 tailwatchd - chkservd - sending OOM notification
13812 root 6.58 1.75 tailwatchd - chkservd - sending OOM notification


For addtional details, see the attached dmesg log dump.



Preview of “oom_dmesg.txt”
[157770.664508] [13518] 506 13518 64760 8629 1 0 0 php-cgi
[157770.666523] [13519] 506 13519 64778 8721 1 0 0 php-cgi
[157770.668516] [13520] 506 13520 64779 8731 0 0 0 php-cgi
[157770.670589] [13521] 506 13521 64827 8698 0 0 0 php-cgi
[157770.672543] [13522] 506 13522 64518 8451 0 0 0 php-cgi
[157770.674369] [13523] 99 13523 140791 1188 1 0 0 httpd
[157770.676201] [13551] 506 13551 64135 8027 1 0 0 php-cgi
[157770.678169] [13570] 506 13570 62488 6432 0 0 0 php-cgi
[157770.680362] Out of memory: Kill process 12653 (mysqld) score 142 or sacrifice child
[157770.682397] Killed process 12653, UID 496, (mysqld) total-vm:923976kB, anon-rss:272236kB, file-rss:36kB

Following are my cat /etc/my.cnf settings:

s26.postimg.org/wxfyaxjt5/mycnf.jpg

 

[SysSachin]

Out of memory: The process “mysqld” was terminated because the system is low on memory.

Mysql process terminated due to low memory on your server. Please check the memory status using the below command.

free -m

 

[gseven]

This is the current status of RAM:
s26.postimg.org/d5iu8rekp/Screen_Shot_2017-03-25_at_6.23.26_PM.png
It suddenly crashes in 2-3 days and works fine again after restart. Corrupt database could be an issue?

Mysql process terminated due to low memory on your server. Please check the memory status using the below command.

free -m

Mod Note: Please Attach Images to Your Posts

 

[SysSachin]

Hello,

You are facing because of low memory so you need to monitor your server and need to find out which process is using your server
memory. Also I will suggest you please try to run MySQLTuner.pl on your server and adjust your MySQL configuration.

 

[gseven]

Thank you, SysSachin.

These are the recommendations from MySQLTuner:

https://s26.postimg.org/461o4nibt/rec_1.jpg

Hello,

You are facing because of low memory so you need to monitor your server and need to find out which process is using your server
memory. Also I will suggest you please try to run MySQLTuner.pl on your server and adjust your MySQL configuration.

 

[SysSachin]

You can modify/add values to the /etc/my.cnf file based on the "Variables to adjust" from the MySQL tuner. Let MySQL run for at least 24 hours after the change and then run the tuner again to see if additional adjustments are recommended.

Also, Monitor your server and try to find out which process is using high memory on your server

 

[cPanelMichael]

Hello,

You may also want to utilize the "mysqladmin processlist" command to see if any particular databases or tables are using an excessive amount of resources at any given time.

Thank you.

 

[ssfred]

Hello

Since OOM killer frequently terminates MySQL server, it means that MySQL keeps on occupying the top position for resource usage. So the prime target is d to identify the cause for the issue. There can be multiple reasons such as a corrupted DB, poorly optimized db queries, unlimited connections etc. Kindly perform the following checks
1. As mentioned by cPanelMichael, Check the active mysql process by executing the command

#mysqladmin processlist
or
#mysqladmin proc stat

2. Check whether slow queries are eating up resources by enabling "Slow Query Log"

3. Check whether any resource monitoring tools like PRM are installed and they gets triggered and terminates the process.

Thank You,