Mysql 4.1 INNODB crash bug recovery guide
---------------------------------------------------------------
Affected Systems:
Machines that are running mysql 4.1 that had no root password set previously,
or have changed root's homedir,
or have run out of disk space during the update
or an automated process killer has terminated the update process (because of the size of the update this appears to be much more likely to happen)
or the system runs out of memory and the kernel kills mysqld
Document Version: 2.5
WARNINGS:
STOP : If there are any problems or you are uncomfortable doing this procedure, do not proceed and open a ticket @ https://tickets.cpanel.net/submit/
Make sure you have free disk space in /home and /var/lib/mysql or where ever your mysql databases are stored. This guide assumes they are in /var/lib/mysql.
Step 1:
Disable tailwatchd (chkservd on 11.23) to prevent mysql from being restarted. Stop any other processes that may access mysql including munin, backups, and the webserver if possible.
Step 2:
Add
innodb_force_recovery = 3
to /etc/my.cnf
Step 3:
Run
Step 4:
Verify mysql is up
Step 6:
Set a mysql root password
Check to see if they have moved root's homedir. If they have copy the ~root/.my.cnf file to /root/.my.cnf
Support for checking this will be added ASAP.
Step 7:
If you do not have any innodb data you care about you can skip this step.
Download http://koston.org/dumpdbs.txt
Step 8:
If you do not have any innodb data you care about you can skip this step.
Run the script. It should dump the databases using innodb into /home/innodb_dumps/
Step 9:
Stop Mysql
Step 10:
Step 11:
Remove
innodb_force_recovery = 3
from /etc/my.cnf
Step 12:
If you do not have any innodb data you care about you can skip this step.
Restart mysql (
)
Step 13:
If you do not have any innodb data you care about you can skip this step.
For every database that was dummped into /home/innodb_dumps/ we need to move away the databases so they can be restored from the dumps
Next create empty database dirs for the .sql files to be restored into
Step 14:
If you do not have any innodb data you care about you can skip this step.
For every database that was dummped into /home/innodb_dumps/
Step 15:
If you do not have any innodb data you care about you can skip this step.
Verify mysql did not crash during the restore
Step 16:
==========
Note: cPanel keeps backups of your horde and roundcube databases in /var/cpanel/roundcube and /var/cpanel/horde.
and
May be able to be used to get horde or roundcube working in a pinch.
---------------------------------------------------------------
Affected Systems:
Machines that are running mysql 4.1 that had no root password set previously,
or have changed root's homedir,
or have run out of disk space during the update
or an automated process killer has terminated the update process (because of the size of the update this appears to be much more likely to happen)
or the system runs out of memory and the kernel kills mysqld
Document Version: 2.5
WARNINGS:
STOP : If there are any problems or you are uncomfortable doing this procedure, do not proceed and open a ticket @ https://tickets.cpanel.net/submit/
Make sure you have free disk space in /home and /var/lib/mysql or where ever your mysql databases are stored. This guide assumes they are in /var/lib/mysql.
Step 1:
Disable tailwatchd (chkservd on 11.23) to prevent mysql from being restarted. Stop any other processes that may access mysql including munin, backups, and the webserver if possible.
Code:
killall -TERM tailwatchdAdd
innodb_force_recovery = 3
to /etc/my.cnf
Step 3:
Run
Code:
/scripts/restartsrv_mysqlVerify mysql is up
Step 6:
Set a mysql root password
Code:
echo "root RANDOMPASSHERE" | /scripts/mysqlpasswdSupport for checking this will be added ASAP.
Step 7:
If you do not have any innodb data you care about you can skip this step.
Download http://koston.org/dumpdbs.txt
Code:
wget -O dumpdbs.pl [url]http://koston.org/dumpdbs.txt[/url]If you do not have any innodb data you care about you can skip this step.
Run the script. It should dump the databases using innodb into /home/innodb_dumps/
Step 9:
Stop Mysql
Step 10:
Code:
mkdir /var/lib/mysql/INNODB_BACKUPS
mv /var/lib/mysql/ib* /var/lib/mysql/INNODB_BACKUPSRemove
innodb_force_recovery = 3
from /etc/my.cnf
Step 12:
If you do not have any innodb data you care about you can skip this step.
Restart mysql (
Code:
/scripts/restartsrv_mysqlStep 13:
If you do not have any innodb data you care about you can skip this step.
For every database that was dummped into /home/innodb_dumps/ we need to move away the databases so they can be restored from the dumps
Code:
mv /var/lib/mysql/MYSQLDBNAME /var/lib/mysql/INNODB_BACKUPS
Code:
mkdir /var/lib/mysql/MYSQLDBNAME
chown mysql:mysql /var/lib/mysql/MYSQLDBNAMEIf you do not have any innodb data you care about you can skip this step.
Code:
cd /home/innodb_dumps
Code:
mysql DBNAME < DBNAME.sqlIf you do not have any innodb data you care about you can skip this step.
Verify mysql did not crash during the restore
Step 16:
Code:
Restart mysql and tailwatchd (chkservd on 11.23) and any other process you may have stopped.
/scripts/restartsrv_mysql
/scripts/restartsrv_tailwatchd==========
Note: cPanel keeps backups of your horde and roundcube databases in /var/cpanel/roundcube and /var/cpanel/horde.
Code:
/usr/local/cpanel/bin/update-roundcube
Code:
/usr/local/cpanel/bin/update-horde
