chkservd/mysql problem

[junglecat]

WHM 11.1.0 cPanel 11.2.16-C12520
FEDORA 4 i686 - WHM X v3.1.0

Mysql started failing every 9 minutes yesterday. Turns out chksrvd was shutting it down:

070527 10:06:03 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.27-standard' socket: '/var/lib/mysql/mysql.sock' port: 3306 MyS
QL Community Edition - Standard (GPL)
070527 10:14:27 [Note] /usr/sbin/mysqld: Normal shutdown

070527 10:14:29 InnoDB: Starting shutdown...
070527 10:14:30 InnoDB: Shutdown completed; log sequence number 4 1363196629
070527 10:14:30 [Note] /usr/sbin/mysqld: Shutdown complete

070527 10:14:30 mysqld ended

070527 10:14:46 mysqld started
070527 10:14:46 InnoDB: Started; log sequence number 4 1363196629
070527 10:14:46 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.27-standard' socket: '/var/lib/mysql/mysql.sock' port: 3306 MyS
QL Community Edition - Standard (GPL)
070527 10:14:47 [Note] /usr/sbin/mysqld: Normal shutdown

070527 10:14:47 InnoDB: Starting shutdown...
070527 10:14:50 InnoDB: Shutdown completed; log sequence number 4 1363196629
070527 10:14:50 [Note] /usr/sbin/mysqld: Shutdown complete

070527 10:14:50 mysqld ended

070527 10:14:51 mysqld started
070527 10:14:51 InnoDB: Started; log sequence number 4 1363196629
070527 10:14:51 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.27-standard' socket: '/var/lib/mysql/mysql.sock' port: 3306 MyS
QL Community Edition - Standard (GPL)
070527 10:23:15 [Note] /usr/sbin/mysqld: Normal shutdown

070527 10:23:17 InnoDB: Starting shutdown...
070527 10:23:18 InnoDB: Shutdown completed; log sequence number 4 1364079029
070527 10:23:18 [Note] /usr/sbin/mysqld: Shutdown complete

070527 10:23:18 mysqld ended

070527 10:23:34 mysqld started
070527 10:23:34 InnoDB: Started; log sequence number 4 1364079029
070527 10:23:34 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.27-standard' socket: '/var/lib/mysql/mysql.sock' port: 3306 MyS
QL Community Edition - Standard (GPL)
070527 10:23:36 [Note] /usr/sbin/mysqld: Normal shutdown

070527 10:23:36 InnoDB: Starting shutdown...
070527 10:23:39 InnoDB: Shutdown completed; log sequence number 4 1364079029
070527 10:23:39 [Note] /usr/sbin/mysqld: Shutdown complete

070527 10:23:39 mysqld ended

070527 10:23:39 mysqld started
070527 10:23:39 InnoDB: Started; log sequence number 4 1364079029
070527 10:23:39 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.27-standard' socket: '/var/lib/mysql/mysql.sock' port: 3306 MyS
QL Community Edition - Standard (GPL)
070527 10:32:03 [Note] /usr/sbin/mysqld: Normal shutdown

070527 10:32:05 InnoDB: Starting shutdown...
070527 10:32:06 InnoDB: Shutdown completed; log sequence number 4 1364079255
070527 10:32:06 [Note] /usr/sbin/mysqld: Shutdown complete

070527 10:32:06 mysqld ended

070527 10:32:23 mysqld started
InnoDB: Warning: we did not need to do crash recovery, but log scan
InnoDB: progressed past the checkpoint lsn 4 1364079255 up to lsn 4 1364079265
070527 10:32:23 InnoDB: Started; log sequence number 4 1364079255
070527 10:32:23 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.27-standard' socket: '/var/lib/mysql/mysql.sock' port: 3306 MyS
QL Community Edition - Standard (GPL)
070527 10:32:24 [Note] /usr/sbin/mysqld: Normal shutdown

070527 10:32:24 InnoDB: Starting shutdown...
070527 10:32:27 InnoDB: Shutdown completed; log sequence number 4 1364079265
070527 10:32:27 [Note] /usr/sbin/mysqld: Shutdown complete

070527 10:32:27 mysqld ended

070527 10:32:28 mysqld started
070527 10:32:28 InnoDB: Started; log sequence number 4 1364079265
070527 10:32:28 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.27-standard' socket: '/var/lib/mysql/mysql.sock' port: 3306 MyS
QL Community Edition - Standard (GPL)
070527 10:40:53 [Note] /usr/sbin/mysqld: Normal shutdown

070527 10:40:55 InnoDB: Starting shutdown...
070527 10:40:57 InnoDB: Shutdown completed; log sequence number 4 1365845646
070527 10:40:57 [Note] /usr/sbin/mysqld: Shutdown complete

070527 10:40:57 mysqld ended

070527 10:41:14 mysqld started
070527 10:41:14 InnoDB: Started; log sequence number 4 1365845646
070527 10:41:14 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.27-standard' socket: '/var/lib/mysql/mysql.sock' port: 3306 MyS
QL Community Edition - Standard (GPL)
070527 10:41:16 [Note] /usr/sbin/mysqld: Normal shutdown

070527 10:41:18 InnoDB: Starting shutdown...
070527 10:41:20 InnoDB: Shutdown completed; log sequence number 4 1365845646
070527 10:41:20 [Note] /usr/sbin/mysqld: Shutdown complete

070527 10:41:20 mysqld ended

070527 10:41:21 mysqld started
070527 10:41:21 InnoDB: Started; log sequence number 4 1365845646
070527 10:41:21 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.27-standard' socket: '/var/lib/mysql/mysql.sock' port: 3306 MyS
QL Community Edition - Standard (GPL)

When I disabled mysql monitoring, mysql stopped shutting down.

But what is the root cause of the problem? Why would chksrvd shut down mysql? Is it a cpanel bug or something else? I forced a cpanel update this morning, and that didn't fix the problem. I couldn't stop it until I disabled mysql monitoring.

Obviously there is *something* that needs to be fixed, but I don't know how to find out what.

 

[junglecat]

Ok, it's getting serious now.

In cpanel in user's accounts, no databases are showing at all. And when you go into phpmyadmin, this error shows:

Wrong username/password. Access denied

 

[junglecat]

I tried changing the mysql password and that didn't fix it.

 

[junglecat]

Mysql is now completely down and the sites on the server are offline. I cannot get mysql up.

 

[Bulk]

Pardon the bump - I'm getting the same problem now, chkservd is shutting down mysql every 9 minutes and turning off monitoring has stopped the shutdowns - did you (or anyone) ever discover the problem?

 

[junglecat]

It turned out to be a cpanel bug on my server. The cpanel people went in there and fixed it, but I have no idea what they did.