The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

mySQL Fails after mySQL Password Reset

Discussion in 'General Discussion' started by eglwolf, Mar 28, 2011.

  1. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    168
    Likes Received:
    0
    Trophy Points:
    16
    I reset the password on my mysql (5.01) database through WHM. Now mysql won't restart. Any ideas???
     
  2. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    What is the exact error being returned by MySQL when trying to restart it? Also, do you have any entries in /usr/local/cpanel/logs/error_log or in /var/lib/mysql/hostname.err log when you try to restart it? Please replace hostname.err with the server's full hostname for that log file.
     
  3. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    168
    Likes Received:
    0
    Trophy Points:
    16
    I have tried to force a cpanel update, I have tried to force a mysql update. I also change the reset the root password to the server as well as tried to reset the mysql password.

    One tip I saw was to move and rename /var/lib/mysql to /var/lib/mysql.bak and for an mysqlupdate, then I freaked when all my databases disappeared. So I moved the mysql.bak to mysql


    Here is the error message:
    Starting MySQL. ERROR! Manager of pid-file quit without updating file.

    I just restarted my server and here is what is in the /usr/cpanel/log/error_log file:

    Code:
    [2011-03-28 14:23:12 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
    	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
    	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
    [2011-03-28 14:23:12 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
     at /usr/local/cpanel/Cpanel/Mysql.pm line 45
    	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
    	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
    mysqladmin: connect to server at 'localhost' failed
    error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
    Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
    ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
    [2011-03-28 14:27:12 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
    	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
    	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
    [2011-03-28 14:27:12 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
     at /usr/local/cpanel/Cpanel/Mysql.pm line 45
    	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
    	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
    ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
    /usr/bin/mysqladmin: connect to server at 'localhost' failed
    error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
    Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
    [2011-03-28 14:27:42 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
    	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
    	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
    [2011-03-28 14:27:42 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
     at /usr/local/cpanel/Cpanel/Mysql.pm line 45
    	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
    	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
    mysqladmin: connect to server at 'localhost' failed
    error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
    Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
    [2011-03-28 14:30:01 -0400] warn [mysqluserstore] Unable to query MySQL database at localhost at /usr/local/cpanel/bin/mysqluserstore line 37
    ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
    ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
    /usr/bin/mysqladmin: connect to server at 'localhost' failed
    error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
    Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
    [2011-03-28 14:38:42 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
    	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
    	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
    [2011-03-28 14:38:42 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
     at /usr/local/cpanel/Cpanel/Mysql.pm line 45
    	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
    	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
    mysqladmin: connect to server at 'localhost' failed
    error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
    Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
    -- RESTART MARKER (PID 25793 at 1301339011)--
    [2011-03-28 15:03:32 -0400] info [cpanellogd] Unable to load summary files: /var/cpanel/bandwidth/smb.hour, /var/cpanel/bandwidth/smb.5min
    Building global cache for cpanel...Done
    [2011-03-28 15:09:16 -0400] warn [quotacheck] User file '/var/cpanel/users/root' is empty or non-existent. at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 106
    	Cpanel::Config::LoadCpUserFile::_load('root', undef, 0) called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 25
    	Cpanel::Config::LoadCpUserFile::load('root') called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 252
    	Cpanel::Config::LoadCpUserFile::loadcpuserfile('root') called at /scripts/quotacheck line 84
    [2011-03-28 15:09:16 -0400] warn [quotacheck] Failed to load cPanel user file for root at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 28
    	Cpanel::Config::LoadCpUserFile::load('root') called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 252
    	Cpanel::Config::LoadCpUserFile::loadcpuserfile('root') called at /scripts/quotacheck line 84
    [2011-03-28 15:12:08 -0400] warn [securemysql] Unable to determine MySQL data directory from mysqladmin variables. at /usr/local/cpanel/Cpanel/MysqlUtils.pm line 109
    	Cpanel::MysqlUtils::getmysqldir() called at /scripts/securemysql line 140
    [2011-03-28 15:12:18 -0400] info [apache_conf_distiller] 'local' datastore in use (/var/cpanel/conf/apache/local)
    [2011-03-28 15:12:22 -0400] info [realadduser] No password specified for cpanel
    [2011-03-28 15:12:22 -0400] die [realadduser] User cpanel already exists
    [2011-03-28 15:12:22 -0400] info [realadduser] No password specified for ftp
    [2011-03-28 15:12:22 -0400] die [realadduser] User ftp already exists
    -- RESTART MARKER (PID 20457 at 1301339605)--
    Building global cache for cpanel...Done
    /scripts/mysqlpasswd: pass is blank
    /scripts/mysqlpasswd: pass is blank
    /usr/bin/mysqladmin: connect to server at 'localhost' failed
    error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
    Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
    mysqladmin: connect to server at 'localhost' failed
    error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
    Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
    [2011-03-28 15:26:06 -0400] warn [securemysql] Unable to determine MySQL data directory from mysqladmin variables. at /usr/local/cpanel/Cpanel/MysqlUtils.pm line 109
    	Cpanel::MysqlUtils::getmysqldir() called at /scripts/securemysql line 140
    [2011-03-28 15:29:44 -0400] warn [quotacheck] User file '/var/cpanel/users/root' is empty or non-existent. at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 106
    	Cpanel::Config::LoadCpUserFile::_load('root', undef, 0) called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 25
    	Cpanel::Config::LoadCpUserFile::load('root') called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 252
    	Cpanel::Config::LoadCpUserFile::loadcpuserfile('root') called at /scripts/quotacheck line 84
    [2011-03-28 15:29:44 -0400] warn [quotacheck] Failed to load cPanel user file for root at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 28
    	Cpanel::Config::LoadCpUserFile::load('root') called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 252
    	Cpanel::Config::LoadCpUserFile::loadcpuserfile('root') called at /scripts/quotacheck line 84
    [2011-03-28 15:31:15 -0400] info [apache_conf_distiller] 'local' datastore in use (/var/cpanel/conf/apache/local)
    [2011-03-28 15:31:16 -0400] info [realadduser] No password specified for cpanel
    [2011-03-28 15:31:16 -0400] die [realadduser] User cpanel already exists
    [2011-03-28 15:31:16 -0400] info [realadduser] No password specified for ftp
    [2011-03-28 15:31:16 -0400] die [realadduser] User ftp already exists
    -- RESTART MARKER (PID 20208 at 1301340774)--
    Building global cache for cpanel...Done
    -- RESTART MARKER (PID 20235 at 1301341551)--
    Building global cache for cpanel...Done
    
    This is what is the the hostname.err file:
    Code:
    110328 15:45:18  mysqld started
    110328 15:45:18 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
    InnoDB: ##########################################################
    InnoDB:                          WARNING!
    InnoDB: The log sequence number in ibdata files is higher
    InnoDB: than the log sequence number in the ib_logfiles! Are you sure
    InnoDB: you are using the right ib_logfiles to start up the database?
    InnoDB: Log sequence number in ib_logfiles is 0 977688, log
    InnoDB: sequence numbers stamped to ibdata file headers are between
    InnoDB: 589505315 589505315 and 589505315 589505315.
    InnoDB: ##########################################################
    InnoDB: The log sequence number in ibdata files does not match
    InnoDB: the log sequence number in the ib_logfiles!
    110328 15:45:18  InnoDB: Database was not shut down normally!
    InnoDB: Starting crash recovery.
    InnoDB: Reading tablespace information from the .ibd files...
    InnoDB: Restoring possible half-written data pages from the doublewrite
    InnoDB: buffer...
    InnoDB: Error: page n:o stored in the page read in is 538976288, should be 3!
    InnoDB: Database page corruption on disk or a failed
    InnoDB: file read of page 3.
    InnoDB: You may have to recover from a backup.
    110328 15:45:19  InnoDB: Page dump in ascii and hex (16384 bytes):
     len 16384; hex 
     
     ) ;InnoDB: End of page dump 110328 14:38:24  InnoDB: Page checksum 1508866187, prior-to-4.0.14-form checksum 3161919040 InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105 InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651 InnoDB: Page number (if stored to page already) 538976288, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288 InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 3. InnoDB: You may have to recover from a backup. InnoDB: It is also possible that your operating InnoDB: system has corrupted its own file cache InnoDB: and rebooting your computer removes the InnoDB: error. InnoDB: If the corrupt page is an index page InnoDB: you can also try to fix the corruption InnoDB: by dumping, dropping, and reimporting InnoDB: the corrupt table. You can use CHECK InnoDB: TABLE to scan your table for corruption. InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. InnoDB: Ending processing because of a corrupt database page. se page.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ) ;InnoDB: End of page dump
    110328 15:45:19  InnoDB: Page checksum 2747739185, prior-to-4.0.14-form checksum 3161919040
    InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105
    InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651
    InnoDB: Page number (if stored to page already) 538976288,
    InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288
    InnoDB: Database page corruption on disk or a failed
    InnoDB: file read of page 3.
    InnoDB: You may have to recover from a backup.
    InnoDB: It is also possible that your operating
    InnoDB: system has corrupted its own file cache
    InnoDB: and rebooting your computer removes the
    InnoDB: error.
    InnoDB: If the corrupt page is an index page
    InnoDB: you can also try to fix the corruption
    InnoDB: by dumping, dropping, and reimporting
    InnoDB: the corrupt table. You can use CHECK
    InnoDB: TABLE to scan your table for corruption.
    InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
    InnoDB: about forcing recovery.
    InnoDB: Ending processing because of a corrupt database page.
    110328 15:45:19  mysqld ended
    
    110328 15:46:31  mysqld started
    110328 15:46:31 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
    InnoDB: ##########################################################
    InnoDB:                          WARNING!
    InnoDB: The log sequence number in ibdata files is higher
    InnoDB: than the log sequence number in the ib_logfiles! Are you sure
    InnoDB: you are using the right ib_logfiles to start up the database?
    InnoDB: Log sequence number in ib_logfiles is 0 977688, log
    InnoDB: sequence numbers stamped to ibdata file headers are between
    InnoDB: 589505315 589505315 and 589505315 589505315.
    InnoDB: ##########################################################
    InnoDB: The log sequence number in ibdata files does not match
    InnoDB: the log sequence number in the ib_logfiles!
    110328 15:46:31  InnoDB: Database was not shut down normally!
    InnoDB: Starting crash recovery.
    InnoDB: Reading tablespace information from the .ibd files...
    InnoDB: Restoring possible half-written data pages from the doublewrite
    InnoDB: buffer...
    InnoDB: Error: page n:o stored in the page read in is 538976288, should be 3!
    InnoDB: Database page corruption on disk or a failed
    InnoDB: file read of page 3.
    InnoDB: You may have to recover from a backup.
    110328 15:46:32  InnoDB: Page dump in ascii and hex (16384 bytes):
    
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              ) ;InnoDB: End of page dump 110328 14:38:24  InnoDB: Page checksum 1508866187, prior-to-4.0.14-form checksum 3161919040 InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105 InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651 InnoDB: Page number (if stored to page already) 538976288, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288 InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 3. InnoDB: You may have to recover from a backup. InnoDB: It is also possible that your operating InnoDB: system has corrupted its own file cache InnoDB: and rebooting your computer removes the InnoDB: error. InnoDB: If the corrupt page is an index page InnoDB: you can also try to fix the corruption InnoDB: by dumping, dropping, and reimporting InnoDB: the corrupt table. You can use CHECK InnoDB: TABLE to scan your table for corruption. InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. InnoDB: Ending processing because of a corrupt database page. se page.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ) ;InnoDB: End of page dump
    110328 15:46:32  InnoDB: Page checksum 2747739185, prior-to-4.0.14-form checksum 3161919040
    InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105
    InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651
    InnoDB: Page number (if stored to page already) 538976288,
    InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288
    InnoDB: Database page corruption on disk or a failed
    InnoDB: file read of page 3.
    InnoDB: You may have to recover from a backup.
    InnoDB: It is also possible that your operating
    InnoDB: system has corrupted its own file cache
    InnoDB: and rebooting your computer removes the
    InnoDB: error.
    InnoDB: If the corrupt page is an index page
    InnoDB: you can also try to fix the corruption
    InnoDB: by dumping, dropping, and reimporting
    InnoDB: the corrupt table. You can use CHECK
    InnoDB: TABLE to scan your table for corruption.
    InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
    InnoDB: about forcing recovery.
    InnoDB: Ending processing because of a corrupt database page.
    110328 15:46:32  mysqld ended
    
    110328 15:50:57  mysqld started
    110328 15:50:57 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
    InnoDB: ##########################################################
    InnoDB:                          WARNING!
    InnoDB: The log sequence number in ibdata files is higher
    InnoDB: than the log sequence number in the ib_logfiles! Are you sure
    InnoDB: you are using the right ib_logfiles to start up the database?
    InnoDB: Log sequence number in ib_logfiles is 0 977688, log
    InnoDB: sequence numbers stamped to ibdata file headers are between
    InnoDB: 589505315 589505315 and 589505315 589505315.
    InnoDB: ##########################################################
    InnoDB: The log sequence number in ibdata files does not match
    InnoDB: the log sequence number in the ib_logfiles!
    110328 15:50:57  InnoDB: Database was not shut down normally!
    InnoDB: Starting crash recovery.
    InnoDB: Reading tablespace information from the .ibd files...
    InnoDB: Restoring possible half-written data pages from the doublewrite
    InnoDB: buffer...
    InnoDB: Error: page n:o stored in the page read in is 538976288, should be 3!
    InnoDB: Database page corruption on disk or a failed
    InnoDB: file read of page 3.
    InnoDB: You may have to recover from a backup.
    110328 15:50:57  InnoDB: Page dump in ascii and hex (16384 bytes):
     len 16384; hex 
     ) ;InnoDB: End of page dump 110328 14:38:24  InnoDB: Page checksum 1508866187, prior-to-4.0.14-form checksum 3161919040 InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105 InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651 InnoDB: Page number (if stored to page already) 538976288, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288 InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 3. InnoDB: You may have to recover from a backup. InnoDB: It is also possible that your operating InnoDB: system has corrupted its own file cache InnoDB: and rebooting your computer removes the InnoDB: error. InnoDB: If the corrupt page is an index page InnoDB: you can also try to fix the corruption InnoDB: by dumping, dropping, and reimporting InnoDB: the corrupt table. You can use CHECK InnoDB: TABLE to scan your table for corruption. InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. InnoDB: Ending processing because of a corrupt database page. se page.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ) ;InnoDB: End of page dump
    110328 15:50:58  InnoDB: Page checksum 2747739185, prior-to-4.0.14-form checksum 3161919040
    InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105
    InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651
    InnoDB: Page number (if stored to page already) 538976288,
    InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288
    InnoDB: Database page corruption on disk or a failed
    InnoDB: file read of page 3.
    InnoDB: You may have to recover from a backup.
    InnoDB: It is also possible that your operating
    InnoDB: system has corrupted its own file cache
    InnoDB: and rebooting your computer removes the
    InnoDB: error.
    InnoDB: If the corrupt page is an index page
    InnoDB: you can also try to fix the corruption
    InnoDB: by dumping, dropping, and reimporting
    InnoDB: the corrupt table. You can use CHECK
    InnoDB: TABLE to scan your table for corruption.
    InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
    InnoDB: about forcing recovery.
    InnoDB: Ending processing because of a corrupt database page.
    110328 15:50:58  mysqld ended
    
    
     
  4. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Did you make a full backup of MySQL or do you have a backup of MySQL prior to the password change? Right now, the issue isn't the password change but that InnoDB ib_logfiles don't match the ibdata content, which indicates a crash. It's pretty difficult to fix InnoDB corruption, so restoring from a backup would be preferred.

    I am leery of providing any recommendations for InnoDB recovery, since you can majorly mess up your databases if done incorrectly. It would be better to have your datacenter, NOC or provider help with any corruption, or to hire someone to fix it if you do not have sound backups.

    Also, always make backups for MySQL especially when using InnoDB before stopping it, restarting it, changing the password, or upgrading it. I've seen too many instances of InnoDB corruption preventing MySQL to restart, and that is exactly what you are showing right now as the issue.
     
  5. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    168
    Likes Received:
    0
    Trophy Points:
    16
    I do not have full backups of each of the databases for the various sites on the server. I could check with the NOC to see if they have a backup of the VPS server from the last day and use that to restore. could I take the databases from the /var/lib/mysql directory and backup those and use those after a reinstall?
     
  6. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    You cannot use the existing data in /var/lib/mysql because that data contains corrupted InnoDB data. Please contact your NOC about any VPS backups that exist. You will need the backup to be before the password was changed. You might even want to find out if they have multiple days of backups in case the corruption existed and wasn't apparent until the MySQL restart.
     
  7. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    168
    Likes Received:
    0
    Trophy Points:
    16
    Thanks. I am using Wordpress for most of my sites. I use the db-manager plugin for each, and that does a weekly backup of the sites and saves those .sql files. That might be the worse case scenario.

    Also I would like to say that once I did the mv /var/lib/mysql to mysql.bak and did a force update of mysql (when I saw all my db where gone) mysql still did not restart. Does that make a difference? Should it have started?
     
  8. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    MySQL would not restart on moving /var/lib/mysql to /var/lib/mysql.bak unless you created /var/lib/mysql with mysql:mysql ownership and ran /scripts/mysqlup --force, since moving the folders with only a restart doesn't recreate MySQL to exist.
     
  9. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    168
    Likes Received:
    0
    Trophy Points:
    16
    The NOC does not have any backups. So, can you tell me what I need to do to try and get this working again.

    Once I have mysql working again can I use one of the previous databases (one at a time) to see if it has problems which cause mySQL not to start, if so I can remove it and them move to the next one. Maybe I get lucky and fine only one or two that have issues and the others are good?

    I can't believe that this all happened from just resetting the password in WHM...
     
  10. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    You can try disabling innodb to begin with and see if it starts up before moving anything. In your /etc/my.cnf file, put the following entry:

    Code:
    skip-innodb
    Then try to restart MySQL. If it starts up, then it was definitely InnoDB. Do you know which specific databases were using InnoDB?
     
  11. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    168
    Likes Received:
    0
    Trophy Points:
    16
    I did what you asked and still will not restart. my.conf looks like
    Code:
    
    [mysqld]
    skip-innodb
    max_connections = 500
    thread_cache_size = 4
    table_cache = 32K
    query_cache_limit=1M
    query_cache_size=24M
    query_cache_type=1
    key_buffer=48M
    read_buffer_size=128K
    join_buffer_size=128K
    connect_timeout=10
    max_allowed_packet=1M
    max_connect_errors=10
    tmp_table_size=128M
    max_heap_table_size=128M
    log=/var/lib/mysql/general.log
    log-slow-queries=/var/lib/mysql/slow.log
    long_query_time=5
    
    cPanel Error Logs
    Code:
    [2011-03-28 16:30:01 -0400] warn [mysqluserstore] Unable to query MySQL database at localhost at /usr/local/cpanel/bin/mysqluserstore line 37
    mysqladmin: connect to server at 'localhost' failed
    error: 'Access denied for user 'root'@'localhost' (using password: YES)'
    [2011-03-28 16:39:57 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
    	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
    	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
    [2011-03-28 16:39:57 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Access denied for user 'root'@'localhost' (using password: YES) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
     at /usr/local/cpanel/Cpanel/Mysql.pm line 45
    	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
    	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
    [2011-03-28 16:41:27 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
    	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
    	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
    [2011-03-28 16:41:27 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
     at /usr/local/cpanel/Cpanel/Mysql.pm line 45
    	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
    	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
    
    Code:
    110328 16:39:35  mysqld started
    110328 16:39:35 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
    110328 16:39:35 [Note] /usr/sbin/mysqld: ready for connections.
    Version: '5.0.91-community-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Edition (GPL)
    110328 16:41:06 [Note] /usr/sbin/mysqld: Normal shutdown
    
    110328 16:41:14 [Note] /usr/sbin/mysqld: Shutdown complete
    
    110328 16:41:14  mysqld ended
    
    110328 16:41:33  mysqld started
    110328 16:41:33 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
    110328 16:41:33 [Note] /usr/sbin/mysqld: ready for connections.
    Version: '5.0.91-community-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Edition (GPL)
    110328 16:41:45 [Note] /usr/sbin/mysqld: Normal shutdown
    
    110328 16:41:55 [Note] /usr/sbin/mysqld: Shutdown complete
    
    110328 16:41:55  mysqld ended
    
    110328 16:41:56  mysqld started
    110328 16:41:56 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
    110328 16:41:56 [Note] /usr/sbin/mysqld: ready for connections.
    Version: '5.0.91-community-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Edition (GPL)
    
    
     
  12. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    168
    Likes Received:
    0
    Trophy Points:
    16
    OK something buzzer has happened. MySQl started working again... Let me explain what I did. When I first got this VPS I setup the host name to use hostname.net (because I owned the .net version of my domain name). However I forgot that I no longer used that one so I would go into channel and change it to .com. I rebooted the server yesterday and it still came back with .net. and I had not gone to fix it again.

    However while waiting for you to reply I went into the VPS control panel and changed it, then I went into cpanel and changed it and verified the new hostname license.

    Then I noticed that mysql was working again.... coincidence?

    I also noticed that the general.log file within mysql is 2.3 GB. That to me seems large, is that okay or should I do something with that?
     
  13. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    168
    Likes Received:
    0
    Trophy Points:
    16
    Another update: in WHM it still shows mysql as down and I keep getting mysql failed @ Mon Mar 28 17:26:13 2011. A restart was attempted automagically.
     
  14. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Glad that MySQL is now working, although it's strange there were InnoDB errors otherwise on startup attempts unless those were prior errors.

    At this point since MySQL itself is running, can you open up a ticket for us to troubleshoot this further? You can open one in WHM > Support Center > Contact cPanel or using the link in my signature. If you have a ticket ID after opening the ticket, please feel free to post it here.
     
  15. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    168
    Likes Received:
    0
    Trophy Points:
    16
    Tristan, Thanks for all your help. My ticket request ID 1316277.
     
  16. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Of note, I've replied to the ticket. The issue with MySQL working itself was due to the password change not having properly changed likely due to the fact InnoDB corruption does still exist on that machine (and likely prevented a proper change previously). I had to do the following steps while keeping "skip-innodb" in /etc/my.cnf to get the MySQL root password properly changed:

    1. Stopped MySQL

    Code:
    /etc/init.d/mysql stop
    2. Started MySQL in safe mode with skip-grant-tables

    Code:
    mysqld_safe --skip-grant-tables &
    3. Went into MySQL command line:

    Code:
    mysql
    4. Reset password in MySQL command line for mysql table

    Code:
    mysql> \u mysql 
    mysql> update user set password=PASSWORD("passwordhere") where User='root'; 
    mysql> flush privileges; 
    mysql> quit
    Of note, I've replaced the actual password used from /root/.my.cnf with a phony password. You can find your password by running a cat on /root/.my.cnf file if you don't know what it happens to be:

    Code:
    cat /root/.my.cnf
    5. Stopped MySQL from safe mode and started it

    Code:
    /etc/init.d/mysql stop
    /etc/init.d/mysql start
    At that point, mysql command line did work without any further issues. I've kept the skip-innodb option in /etc/my.cnf due to the fact that commenting it out won't let MySQL start still due to the existing InnoDB engine table corruption.
     
Loading...

Share This Page