MySQL Randomly Fails then Recovers

[jabelone]

Hi,

I'm running a few websites and databases on a VPS I manage. It's running CentOS 6.6 and has been doing it for a while but it's starting to get annoying. :/ MySQL will randomly just "fail" then be recovered. Most sites are WordPress based and I have a mobile app I created that relies on accessing a database so it's quite important. The problem is I have no idea what is causing this to happen. Here is an example email I get from the server every time it happens.

[B]Server:[/B] gallifrey.********.com.au
[B]Primary IP Address:[/B] ***.***.***.***
[B]Service Name:[/B] mysql
[B]Service Status:[/B] failed
[B]Notification:[/B] The service “mysql” appears to be down.
[B]Service Check Method:[/B] The system’s command to check or to restart this service failed.
[B]Number of Restart Attempts:[/B] 1
[B]Startup Log:[/B]
Starting MySQL...... SUCCESS!
[B]Log Messages:[/B]
150617 12:09:47 mysqld_safe mysqld from pid file /var/lib/mysql/gallifrey.********.com.au.pid ended
150617 12:09:47 [Note] /usr/sbin/mysqld: Shutdown complete
150617 12:09:46 mysqld_safe mysqld restarted
150617 12:09:46 mysqld_safe Number of processes running now: 0
150617 3:26:20 [Note] /usr/sbin/mysqld: ready for connections.
150617 03:26:15 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150617 03:25:04 mysqld_safe mysqld from pid file /var/lib/mysql/gallifrey.********.com.au.pid ended
150617 3:25:04 [Note] /usr/sbin/mysqld: Shutdown complete
150617 03:25:03 mysqld_safe mysqld restarted
150617 03:25:03 mysqld_safe Number of processes running now: 0
150616 12:13:12 [Note] /usr/sbin/mysqld: ready for connections.
150616 12:13:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150616 12:09:44 mysqld_safe mysqld from pid file /var/lib/mysql/gallifrey.********.com.au.pid ended
150616 12:09:44 [Note] /usr/sbin/mysqld: Shutdown complete
150616 12:09:43 mysqld_safe mysqld restarted
150616 12:09:43 mysqld_safe Number of processes running now: 0
[B]Memory Information:[/B]
[LIST]
[*][LIST]
[*]Used: 303 MB
[*]Available: 313 MB
[*]Installed: 616 MB
[/LIST]
[/LIST]
[INDENT]Load Information: 0.20 0.30 0.15
Uptime: 7 days, 19 hours, 33 minutes, and 9 seconds
IOStat Information:
avg-cpu: %user %nice %system %iowait %steal %idle
2.43 0.44 0.65 3.89 0.03 92.56
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
vda 11.58 521.90 115.61 351719556 77913264
ChkServd Version: 15.9[/INDENT]

Then I get one of these:

[B]Server:[/B] gallifrey.********.com.au
[B]Primary IP Address:[/B] ***.***.***.***
[B]Service Name:[/B] mysql
[B]Service Status:[/B] recovered
[B]Notification:[/B] The service “mysql” is now operational.
[B]Service Check Raw Output:[/B]
The 'mysql' service passed the check.
[B]Startup Log:[/B]
Starting MySQL...... SUCCESS!
[B]Log Messages:[/B]
150617 3:26:20 [Note] /usr/sbin/mysqld: ready for connections.
150617 03:26:15 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150617 03:25:04 mysqld_safe mysqld from pid file /var/lib/mysql/gallifrey.********.com.au.pid ended
150617 3:25:04 [Note] /usr/sbin/mysqld: Shutdown complete
150617 03:25:03 mysqld_safe mysqld restarted
150617 03:25:03 mysqld_safe Number of processes running now: 0
150616 12:13:12 [Note] /usr/sbin/mysqld: ready for connections.
150616 12:13:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150616 12:09:44 mysqld_safe mysqld from pid file /var/lib/mysql/gallifrey.********.com.au.pid ended
150616 12:09:44 [Note] /usr/sbin/mysqld: Shutdown complete
150616 12:09:43 mysqld_safe mysqld restarted
150616 12:09:43 mysqld_safe Number of processes running now: 0
150616 3:24:52 [Note] /usr/sbin/mysqld: ready for connections.
[B]Memory Information:[/B]
[LIST]
[*][LIST]
[*]Used: 308 MB
[*]Available: 308 MB
[*]Installed: 616 MB
[/LIST]
[/LIST]
[INDENT]Load Information: 0.22 0.61 0.40
Uptime: 7 days, 10 hours, 51 minutes, and 16 seconds
IOStat Information:
avg-cpu: %user %nice %system %iowait %steal %idle
2.45 0.44 0.65 3.95 0.03 92.47
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
vda 11.68 524.46 117.45 337052860 75479880
ChkServd Version: 15.9

 

[cPanelMichael]

Hello

Do you notice any additional messages in the MySQL error log (/var/lib/mysql/$hostname.err) at the time of the failures?

Thank you.

 

[jabelone]

Hi, sorry for the late reply - for some reason the forum didn't email me when you replied.

Here is the last bit of the log from this morning, when it happened again. I have included a lot of the "before it crashed" if that helps. I got a cPanel service monitor email saying it had failed right around 11:22.

P.S. Sorry for the terrible formatting in the OP, I don't know what happened and it won't let me edit it.


- Removed - No Need For That Much Output in a Thread -

 

[jabelone]

This has been resolved after opening a support ticket with cPanel. As can be seen by these two entries in the log file, the server was running out of memory.

Jul 14 18:50:15 gallifrey kernel: Out of memory: Kill process 22589 (mysqld) score 159 or sacrifice child
Jul 14 18:50:15 gallifrey kernel: Killed process 22589, UID 498, (mysqld) total-vm:1349600kB, anon-rss:161992kB, file-rss:244kB

 

[cPanelMichael]

I'm happy to see the issue is now resolved. Thank you for updating us with the outcome.