SQL Server crashing and restarting, log file attached, what might be the solution?

[maestroc]

The MySQL server on my CentOS cpanel account keeps crashing/restarting at random intervals. The most it ever seems to run is for a day or two. Pasted below is the tail of my sql error log. Anyone have any idea of what I need to adjust or try to repair in order to get the MySQL server to run reliably?

As you can see from the first few lines I have had issues occasionally with crashed tables in various databases. I go in through phpmyadmin and repair them but then they crash again...

Any help would be greatly appreciated. As you can tell I am not incredibly experienced in this area so please speak slowly for me.

140605 14:49:54 [ERROR] /usr/sbin/mysqld: Table './eximstats/sends' is marked as crashed and should be repaired
140605 14:49:54 [ERROR] /usr/sbin/mysqld: Table './eximstats/sends' is marked as crashed and should be repaired
140605 14:50:00 [ERROR] /usr/sbin/mysqld: Table './eximstats/sends' is marked as crashed and should be repaired
140605 14:50:01 [ERROR] /usr/sbin/mysqld: Table './eximstats/sends' is marked as crashed and should be repaired
140605 14:50:02 [ERROR] /usr/sbin/mysqld: Table './eximstats/sends' is marked as crashed and should be repaired
140605 14:50:04 [ERROR] /usr/sbin/mysqld: Table './eximstats/sends' is marked as crashed and should be repaired
140605 14:50:05 [ERROR] /usr/sbin/mysqld: Table './eximstats/sends' is marked as crashed and should be repaired
140605 14:50:06 [ERROR] /usr/sbin/mysqld: Table './eximstats/sends' is marked as crashed and should be repaired
140605 14:50:23 [ERROR] /usr/sbin/mysqld: Table './eximstats/sends' is marked as crashed and should be repaired
140605 14:50:23 [ERROR] /usr/sbin/mysqld: Table './eximstats/sends' is marked as crashed and should be repaired
140605 14:50:24 [ERROR] /usr/sbin/mysqld: Table './eximstats/sends' is marked as crashed and should be repaired
140605 14:50:35 [ERROR] /usr/sbin/mysqld: Table './eximstats/sends' is marked as crashed and should be repaired
140605 14:50:38 [ERROR] /usr/sbin/mysqld: Table './eximstats/sends' is marked as crashed and should be repaired
140605 14:50:42 [ERROR] /usr/sbin/mysqld: Table './eximstats/sends' is marked as crashed and should be repaired
140605 14:50:43 [ERROR] /usr/sbin/mysqld: Table './eximstats/sends' is marked as crashed and should be repaired
140606 04:19:09 mysqld_safe Number of processes running now: 0
140606 04:19:09 mysqld_safe mysqld restarted
140606  4:19:09 [Note] Plugin 'FEDERATED' is disabled.
140606  4:19:09  InnoDB: Initializing buffer pool, size = 110.0M
140606  4:19:09  InnoDB: Completed initialization of buffer pool
InnoDB: Log scan progressed past the checkpoint lsn 6 576517682
140606  4:19:09  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 6 576538928
140606  4:19:10  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
140606  4:19:10  InnoDB: Started; log sequence number 6 576538928
140606  4:19:10 [Note] Event Scheduler: Loaded 0 events
140606  4:19:10 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.73-cll'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPLv2)
140607 01:03:18 mysqld_safe Number of processes running now: 0
140607 01:03:18 mysqld_safe mysqld restarted
140607  1:03:18 [Note] Plugin 'FEDERATED' is disabled.
140607  1:03:18  InnoDB: Initializing buffer pool, size = 110.0M
140607  1:03:18  InnoDB: Completed initialization of buffer pool
InnoDB: Log scan progressed past the checkpoint lsn 6 821854091
140607  1:03:18  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 6 821916080
140607  1:03:18  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
140607  1:03:19  InnoDB: Started; log sequence number 6 821916080
140607  1:03:19 [Note] Event Scheduler: Loaded 0 events
140607  1:03:19 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.73-cll'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPLv2)
140608 21:47:33 mysqld_safe Number of processes running now: 0
140608 21:47:33 mysqld_safe mysqld restarted
140608 21:47:33 [Note] Plugin 'FEDERATED' is disabled.
140608 21:47:33  InnoDB: Initializing buffer pool, size = 110.0M
140608 21:47:33  InnoDB: Completed initialization of buffer pool
InnoDB: Log scan progressed past the checkpoint lsn 6 1207548833
140608 21:47:33  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 6 1207579994
InnoDB: 8 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 9 row operations to undo
InnoDB: Trx id counter is 0 35638528
140608 21:47:33  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Starting in background the rollback of uncommitted transactions
140608 21:47:34  InnoDB: Rolling back trx with id 0 35638155, 1 rows to undo
140608 21:47:34  InnoDB: Started; log sequence number 6 1207579994

InnoDB: Rolling back of trx id 0 35638155 completed
140608 21:47:34  InnoDB: Rolling back trx with id 0 35638153, 1 rows to undo

InnoDB: Rolling back of trx id 0 35638153 completed
140608 21:47:34  InnoDB: Rolling back trx with id 0 35638136, 1 rows to undo

InnoDB: Rolling back of trx id 0 35638136 completed
140608 21:47:34  InnoDB: Rolling back trx with id 0 35638135, 1 rows to undo

InnoDB: Rolling back of trx id 0 35638135 completed
140608 21:47:34  InnoDB: Rolling back trx with id 0 35638134, 1 rows to undo

InnoDB: Rolling back of trx id 0 35638134 completed
140608 21:47:34  InnoDB: Rolling back trx with id 0 35638123, 2 rows to undo

InnoDB: Rolling back of trx id 0 35638123 completed
140608 21:47:34  InnoDB: Rolling back trx with id 0 35638116, 1 rows to undo

InnoDB: Rolling back of trx id 0 35638116 completed
140608 21:47:34  InnoDB: Rolling back trx with id 0 35638115, 1 rows to undo

InnoDB: Rolling back of trx id 0 35638115 completed
140608 21:47:34  InnoDB: Rollback of non-prepared transactions completed
140608 21:47:34 [Note] Event Scheduler: Loaded 0 events
140608 21:47:34 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.73-cll'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPLv2)

 

[cPanelMichael]

Hello

It looks like there is InnoDB corruption based on the error output. Could you post the entries added to your /etc/my.cnf file?

Thank you.

 

[maestroc]

Hello

It looks like there is InnoDB corruption based on the error output. Could you post the entries added to your /etc/my.cnf file?

Thank you.

Here it is:

[mysqld]
innodb_file_per_table=1
local-infile=0
open_files_limit=22164
query_cache_type=1
query_cache_size=90M
query_cache_limit=100M
tmp_table_size=156M
max_heap_table_size=156M
thread_cache_size=16K
key_buffer_size=64M
join_buffer_size=900K
max_connections=200
innodb_buffer_pool_size=110M
table_cache=3K
table_definition_cache=4K

 

[cPanelMichael]

The following document is a good place to start when handling InnoDB corruption:

MySQL :: MySQL 5.5 Reference Manual :: 14.19 InnoDB Backup and Recovery

Thank you.

 

[maestroc]

So it sounds like I need to drop and recreate that database from a backup. If this was for a normal account I would know how to do that with no problem (I've done it before, deleting an account then restoring it from backup), but since this is in reference to the mailserver tables I am not familiar with how to do that part of things. Can it be done through WHM as I have done with the others?

Also will doing so screw up the way mail is processed while the deletion/restore process is carried out?

Like I said, I'm new to this part of server management...

 

[cPanelMichael]

The references in the error logs to the eximstats database is not related to the InnoDB corruption. Those are separate, unrelated, error messages in the MySQL error log.

Thank you.

 

[maestroc]

Like I said, I am kind of clueless here. In the help file you referenced it talks like the only way to fix the corruption is to restore the database from a backup. If the eximstats db is not the culprit how do I figure out what is doing it or what should my steps be to fixing the issue?

 

[cPanelMichael]

I don't recommend attempting InnoDB recovery if you are not comfortable doing so. Instead, you may want to consult with a qualified system administrator for assistance with this particular issue.

Thank you.