Community Forums
Connect with us on LinkedIn
+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 15 of 17
  1. #1
    cPanel Staff cpanelnick's Avatar
    Join Date
    Feb 2003
    Location
    Houston, TX
    Posts
    4,597

    Default MySQL InnoDB crash recovery guide for 11.23,11.24 (mysql will not start)

    Mysql 4.1 INNODB crash bug recovery guide
    ---------------------------------------------------------------

    Affected Systems:
    Machines that are running mysql 4.1 that had no root password set previously,
    or have changed root's homedir,
    or have run out of disk space during the update
    or an automated process killer has terminated the update process (because of the size of the update this appears to be much more likely to happen)
    or the system runs out of memory and the kernel kills mysqld


    Document Version: 2.5

    WARNINGS:
    STOP : If there are any problems or you are uncomfortable doing this procedure, do not proceed and open a ticket @ https://tickets.cpanel.net/submit/


    Make sure you have free disk space in /home and /var/lib/mysql or where ever your mysql databases are stored. This guide assumes they are in /var/lib/mysql.

    Step 1:
    Disable tailwatchd (chkservd on 11.23) to prevent mysql from being restarted. Stop any other processes that may access mysql including munin, backups, and the webserver if possible.
    Code:
    killall -TERM tailwatchd
    Step 2:
    Add
    innodb_force_recovery = 3
    to /etc/my.cnf

    Step 3:
    Run
    Code:
    /scripts/restartsrv_mysql
    Step 4:
    Verify mysql is up

    Step 6:
    Set a mysql root password
    Code:
    echo "root RANDOMPASSHERE" | /scripts/mysqlpasswd
    Check to see if they have moved root's homedir. If they have copy the ~root/.my.cnf file to /root/.my.cnf
    Support for checking this will be added ASAP.


    Step 7:
    If you do not have any innodb data you care about you can skip this step.


    Download http://koston.org/dumpdbs.txt
    Code:
    wget -O dumpdbs.pl http://koston.org/dumpdbs.txt
    Step 8:
    If you do not have any innodb data you care about you can skip this step.

    Run the script. It should dump the databases using innodb into /home/innodb_dumps/

    Step 9:
    Stop Mysql

    Step 10:
    Code:
    mkdir /var/lib/mysql/INNODB_BACKUPS
    	mv /var/lib/mysql/ib* /var/lib/mysql/INNODB_BACKUPS
    Step 11:
    Remove
    innodb_force_recovery = 3
    from /etc/my.cnf

    Step 12:
    If you do not have any innodb data you care about you can skip this step.

    Restart mysql (
    Code:
    /scripts/restartsrv_mysql
    )


    Step 13:
    If you do not have any innodb data you care about you can skip this step.

    For every database that was dummped into /home/innodb_dumps/ we need to move away the databases so they can be restored from the dumps
    Code:
    mv /var/lib/mysql/MYSQLDBNAME /var/lib/mysql/INNODB_BACKUPS
    Next create empty database dirs for the .sql files to be restored into
    Code:
    mkdir /var/lib/mysql/MYSQLDBNAME
    	  chown mysql:mysql /var/lib/mysql/MYSQLDBNAME
    Step 14:
    If you do not have any innodb data you care about you can skip this step.

    Code:
    cd /home/innodb_dumps
    For every database that was dummped into /home/innodb_dumps/

    Code:
    mysql DBNAME < DBNAME.sql
    Step 15:
    If you do not have any innodb data you care about you can skip this step.

    Verify mysql did not crash during the restore

    Step 16:
    Code:
    Restart mysql and tailwatchd (chkservd on 11.23) and any other process you may have stopped.
      /scripts/restartsrv_mysql
      /scripts/restartsrv_tailwatchd


    ==========

    Note: cPanel keeps backups of your horde and roundcube databases in /var/cpanel/roundcube and /var/cpanel/horde.

    Code:
    /usr/local/cpanel/bin/update-roundcube
    and
    Code:
    /usr/local/cpanel/bin/update-horde
    May be able to be used to get horde or roundcube working in a pinch.
    -Nick
    cPanel Inc.

    Need support? Submit a request here. Complimentary support is available to all license holders regardless of where you purchased your license.
    Need a complimentary support account? Create one here.

  2. #2
    cPanel Staff cpanelnick's Avatar
    Join Date
    Feb 2003
    Location
    Houston, TX
    Posts
    4,597

    Default

    This space held for future posts
    -Nick
    cPanel Inc.

    Need support? Submit a request here. Complimentary support is available to all license holders regardless of where you purchased your license.
    Need a complimentary support account? Create one here.

  3. #3
    cPanel Staff cpanelnick's Avatar
    Join Date
    Feb 2003
    Location
    Houston, TX
    Posts
    4,597

    Default

    This space held for future posts (2)
    -Nick
    cPanel Inc.

    Need support? Submit a request here. Complimentary support is available to all license holders regardless of where you purchased your license.
    Need a complimentary support account? Create one here.

  4. #4
    dxm
    dxm is offline
    Member
    Join Date
    Aug 2007
    Posts
    21

    Default

    This worked for me.
    mysql 4.1.22-standard,
    cPanel 11.24.4-R32470 - WHM 11.24.2 - X 3.9
    CENTOS 4.7 i686 on virtuozzo

    Latest update ran and mysql wouldn't start, giving innodb errors.

    This process has allowed me to get my innodb databases back up and running.
    Sigh of relief all round!

    Thanks.

  5. #5
    Member
    Join Date
    Apr 2005
    Posts
    6

    Default But if I using InnoDB?

    Everywhere You write
    If you do not have any innodb data you care about you can skip this step.
    But I have Innodb? how can enable it?

  6. #6
    Member
    Join Date
    Jun 2004
    Posts
    32

    Angry ... ahi ahi

    More than 20 accounts with data losted!!!!!

    How can we restore theme?


  7. #7
    Member
    Join Date
    Jul 2005
    Location
    Sticky On Internet
    Posts
    555

    Default

    Quote Originally Posted by Tigran View Post
    Everywhere You write
    If you do not have any innodb data you care about you can skip this step.
    But I have Innodb? how can enable it?
    If you do not have any innodb data you care about you can skip this step.
    If you DO NOT have any innodb data you care about, you can skip this step.

    so if you have, make sure to follow the steps and advice.
    Learn atleast A word Daily

    7+1 Dedicated Boxes with cPanel...

  8. #8
    cPanel Staff cpanelnick's Avatar
    Join Date
    Feb 2003
    Location
    Houston, TX
    Posts
    4,597

    Default

    Quote Originally Posted by phoenixweb View Post
    More than 20 accounts with data losted!!!!!

    How can we restore theme?

    If there are any problems or you are uncomfortable doing this procedure, do not proceed and open a ticket @
    https://tickets.cpanel.net/submit/
    -Nick
    cPanel Inc.

    Need support? Submit a request here. Complimentary support is available to all license holders regardless of where you purchased your license.
    Need a complimentary support account? Create one here.

  9. #9
    Member
    Join Date
    Jun 2004
    Posts
    32

    Unhappy

    ... this procedure was right to restart MySql.

    However we advise of the problem when was too late, MySql run out of space on his partition due to too many restart / logs / shits, and this beginning to corrupt tables over tables.

    So after this recovery we had to recovery also the database from the last backup (15 of december)....

    not good.

    This is the FIRST time that we loose costumer's data since 2003.

    Guys... that was a very bad joke for us... i hope this is the last time that you release a major upgrade without make acid test over several machines and configurations.

    Very bad.
    Please test it before provide the upgrades.

    Bye

  10. #10
    cPanel Partner NOC cPanel Partner NOC Badge
    Join Date
    May 2003
    Posts
    92
    cPanel/Enkompass Access Level

    DataCenter Provider

    Default

    While this is a serious issue, I am surprised that people still do not set root passwords on MySQL or keep vigilante for available disk space. These are common system administration tasks. cPanel cannot take full blame for this issue (although it would have been nice to have a bit more notice of the release in preparation for it)...
    Last edited by movielad; 12-19-2008 at 06:41 AM.
    Memset Dedicated Hosting
    Managed, dedicated and VPS systems
    http://www.memset.com

  11. #11
    Member
    Join Date
    Jun 2004
    Posts
    32

    Exclamation

    Quote Originally Posted by movielad View Post
    While this is a serious issue, I am surprised that people still do not set root passwords on MySQL or keep vigilante for available disk space. These are common system administration tasks. cPanel cannot take full blame for this issue (although it would have been nice to have a bit more notice of the release in preparation for it)...

    Dear Movielad,

    we keep always monitored disk space, but when yesterday MySql crashed, it beginning to eat all disk space and end it in less than 3 hours... Since we run update in the night 'cause it's the time with minimum level of traffic over the server, we cannot notice this on time to fix the problem before it was too late.

    We always take a look at the status of the server and read every email sended by the monitor of Cpanel (like DISK SPACE USAGE WARNING), and always take care about this things.

    BUT, this time was strictly related to the update and happened too fast and in the night so it give us no a chance to be ready for this.

    I'm planning to auto-stop the mysql server when our monitor agent track that the Mysql partition is running out of space. However this will be my way to avoid the problem but should be implemented directly by Cpanel system and not by a monitor selfdeveloped by our staff.

    Bye
    Max

  12. #12
    cPanel Staff cpanelnick's Avatar
    Join Date
    Feb 2003
    Location
    Houston, TX
    Posts
    4,597

    Default

    Quote Originally Posted by phoenixweb View Post
    Dear Movielad,

    we keep always monitored disk space, but when yesterday MySql crashed, it beginning to eat all disk space and end it in less than 3 hours... Since we run update in the night 'cause it's the time with minimum level of traffic over the server, we cannot notice this on time to fix the problem before it was too late.

    We always take a look at the status of the server and read every email sended by the monitor of Cpanel (like DISK SPACE USAGE WARNING), and always take care about this things.

    BUT, this time was strictly related to the update and happened too fast and in the night so it give us no a chance to be ready for this.

    I'm planning to auto-stop the mysql server when our monitor agent track that the Mysql partition is running out of space. However this will be my way to avoid the problem but should be implemented directly by Cpanel system and not by a monitor selfdeveloped by our staff.

    Bye
    Max
    Max,

    If you can just upgrade to MySQL 5.0.x as its InnoDB support is much more reliable.
    -Nick
    cPanel Inc.

    Need support? Submit a request here. Complimentary support is available to all license holders regardless of where you purchased your license.
    Need a complimentary support account? Create one here.

  13. #13
    Member
    Join Date
    Jun 2004
    Posts
    32

    Default

    But:

    1) Not only INNODB crashed... running out of space destroyed also MyISAM tables. By the way INNODB was almost completely restored with your recovery guide... the problem was not the secondary cause-effect of MySql behaviour.

    Upgrading to a better handling of INNODB should not save me from this kind of problem.

    2) I cannot upgrade, since we noticed that old's script website doens't support the new query style of LEFT JOIN SYNTAX of the Mysql 5.2 version.

    We have very few server with 4.1 installed over but cannot upgrade 'cause our costumer will kill our staff in less than 15 minuts.

  14. #14
    Member LiNUxG0d's Avatar
    Join Date
    Jun 2003
    Location
    Gatineau, Quebec, Canada
    Posts
    206

    Default

    Do we have any suggestions for when the fix doesn't fix?

    I ran it word for word and when I get to step 3, mysql just doesn't start and it just loads my err log up with tons of garbage.

    This is not very cool.
    http://www.okteck.com/
    The best web hosting, reseller hosting and dedicated server packages!

  15. #15
    Technical Product Specialist cPanelDavidG's Avatar
    Join Date
    Nov 2006
    Location
    Houston, TX
    Posts
    11,189
    cPanel/Enkompass Access Level

    Root Administrator

    Default

    Quote Originally Posted by LiNUxG0d View Post
    Do we have any suggestions for when the fix doesn't fix?

    I ran it word for word and when I get to step 3, mysql just doesn't start and it just loads my err log up with tons of garbage.

    This is not very cool.
    Do not proceed. Immediately submit a support ticket so our technical analysts can assist you with this: http://tickets.cPanel.net/submit

Similar Threads & Tags
Similar threads

  1. Replies: 4
    Last Post: 10-10-2010, 02:55 PM
  2. Replies: 4
    Last Post: 10-10-2010, 02:55 PM
  3. MySQL 5.1.40 InnoDB
    By yuminanako in forum cPanel and WHM Discussions
    Replies: 1
    Last Post: 12-28-2009, 09:54 AM
  4. Mysql recovery
    By teletele in forum New User Questions
    Replies: 0
    Last Post: 03-30-2005, 08:33 AM
  5. Has anyone implemented InnoDB in MySQL?
    By jsteel in forum cPanel and WHM Discussions
    Replies: 5
    Last Post: 02-13-2003, 11:17 PM
Linkedin       Facebook       Twitter       RSS       Flickr       YouTube