MySQL randomly restarting

[mcn_doug]

We have a little bit older cPanel server (running cPanel 11.25.0-R46156 - WHM 11.25.0 - X 3.9 ), and it is randomly restarting the mysql service.

Last Friday, the MySQL service restarted 2 times (approx 5:00 and 9:00 in the am). Yesterday, it restarted 2 times (approximately 9:00 in the am, and 3:00 in the afternoon).

We are not able to find any useful information about why the service is randomly restarting on this one server. When we look at /usr/local/sim/sim/log it shows:

[06/07/10 15:00:02]: MYSQL service is offline.
[06/07/10 15:00:02]: Restarted MYSQL service (2 MYSQL events today).

When I look at the mysql server error log (/var/lib/mysql/sername.err), it shows:

100607 14:56:05 [Note] /usr/sbin/mysqld: Normal shutdown

100607 14:56:07  InnoDB: Starting shutdown...
100607 14:56:08  InnoDB: Shutdown completed; log sequence number 0 967348092
100607 14:56:08 [Note] /usr/sbin/mysqld: Shutdown complete

100607 14:56:08  mysqld ended

100607 14:56:09  mysqld started

Is there some other logs that we can look at, or enable, that will help us to figure out why this is happening? Is there some other kind of monitoring that we can install that will help us find this issue?

Any help with this would be greatly appreciated!

Doug

[garrettp]

When you say you're using SIM, I assume you mean this.

That app does have automatic restart functionality built in, and although I've never used it, I would check to make sure you don't have any settings enabled for restarting the MySQL daemon based on an arbitrary threshold such as number of connections or response time.

[mcn_doug]

Yes, System Integrity Monitor is what I meant, sorry for being unclear.

We saw that SIM had the ability to restart services. We don't have a problem with it restarting things, if it is necessary, but we are wanting to know why it is happening.

We are trying to determine, if it is SIM that is restarting things, why does it think that mysqld needs to be restarted? The sim logs don't really give any useful information for this purpose.

Looking at other log files, our web application is running just fine up to the point where mysql is getting restarted, and then after MySQL is again running. The only reason for any problems with our application, is because mysql is being restarted. It runs without issues before and after the restart.

The system load is not very high (0.5 or less). The server has 4GB of ram, with about 3GB free, so it shouldn't be a memory issue. Munin is enabled, and none of the charts show anything out of the ordinary at the times when MySQL is being restarted.

Any other thoughts of things that we can look into to try and determine why this is happening?

[garrettp]

Check SIM settings as suggested? (I've never used it)

[mcn_doug]

Garret,

Thanks for the suggestion, but we have been through the configuration for SIM, and we are not able to determine how to update the information it is giving us, without rewriting the mysql.chk script that SIM is using.

It appears to just be a shell script, so that might be our next step, but I was not wanting to start rewriting tools, if there was something or somewhere else that we should have been checking.

[garrettp]

Garret,

Thanks for the suggestion, but we have been through the configuration for SIM, and we are not able to determine how to update the information it is giving us, without rewriting the mysql.chk script that SIM is using.

It appears to just be a shell script, so that might be our next step, but I was not wanting to start rewriting tools, if there was something or somewhere else that we should have been checking.

I had a quick look through the SIM config and it doesn't appear that there is anything in there that would restart MySQL based on the thresholds I was mentioning.

With that said, try eliminating SIM as a source of the random restarts by temporarily disabling it's cron job and letting the time pass when these random restarts usually occur. You can toggle the cron job off/on using (according to the README):

# /usr/local/sbin/sim -j