mySQL Fails after mySQL Password Reset

[eglwolf]

I reset the password on my mysql (5.01) database through WHM. Now mysql won't restart. Any ideas???

[cPanelTristan]

What is the exact error being returned by MySQL when trying to restart it? Also, do you have any entries in /usr/local/cpanel/logs/error_log or in /var/lib/mysql/hostname.err log when you try to restart it? Please replace hostname.err with the server's full hostname for that log file.

[eglwolf]

I have tried to force a cpanel update, I have tried to force a mysql update. I also change the reset the root password to the server as well as tried to reset the mysql password.

One tip I saw was to move and rename /var/lib/mysql to /var/lib/mysql.bak and for an mysqlupdate, then I freaked when all my databases disappeared. So I moved the mysql.bak to mysql


Here is the error message:
Starting MySQL. ERROR! Manager of pid-file quit without updating file.

I just restarted my server and here is what is in the /usr/cpanel/log/error_log file:

[2011-03-28 14:23:12 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
[2011-03-28 14:23:12 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
 at /usr/local/cpanel/Cpanel/Mysql.pm line 45
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
[2011-03-28 14:27:12 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
[2011-03-28 14:27:12 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
 at /usr/local/cpanel/Cpanel/Mysql.pm line 45
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
/usr/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
[2011-03-28 14:27:42 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
[2011-03-28 14:27:42 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
 at /usr/local/cpanel/Cpanel/Mysql.pm line 45
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
[2011-03-28 14:30:01 -0400] warn [mysqluserstore] Unable to query MySQL database at localhost at /usr/local/cpanel/bin/mysqluserstore line 37
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
/usr/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
[2011-03-28 14:38:42 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
[2011-03-28 14:38:42 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
 at /usr/local/cpanel/Cpanel/Mysql.pm line 45
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
-- RESTART MARKER (PID 25793 at 1301339011)--
[2011-03-28 15:03:32 -0400] info [cpanellogd] Unable to load summary files: /var/cpanel/bandwidth/smb.hour, /var/cpanel/bandwidth/smb.5min
Building global cache for cpanel...Done
[2011-03-28 15:09:16 -0400] warn [quotacheck] User file '/var/cpanel/users/root' is empty or non-existent. at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 106
	Cpanel::Config::LoadCpUserFile::_load('root', undef, 0) called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 25
	Cpanel::Config::LoadCpUserFile::load('root') called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 252
	Cpanel::Config::LoadCpUserFile::loadcpuserfile('root') called at /scripts/quotacheck line 84
[2011-03-28 15:09:16 -0400] warn [quotacheck] Failed to load cPanel user file for root at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 28
	Cpanel::Config::LoadCpUserFile::load('root') called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 252
	Cpanel::Config::LoadCpUserFile::loadcpuserfile('root') called at /scripts/quotacheck line 84
[2011-03-28 15:12:08 -0400] warn [securemysql] Unable to determine MySQL data directory from mysqladmin variables. at /usr/local/cpanel/Cpanel/MysqlUtils.pm line 109
	Cpanel::MysqlUtils::getmysqldir() called at /scripts/securemysql line 140
[2011-03-28 15:12:18 -0400] info [apache_conf_distiller] 'local' datastore in use (/var/cpanel/conf/apache/local)
[2011-03-28 15:12:22 -0400] info [realadduser] No password specified for cpanel
[2011-03-28 15:12:22 -0400] die [realadduser] User cpanel already exists
[2011-03-28 15:12:22 -0400] info [realadduser] No password specified for ftp
[2011-03-28 15:12:22 -0400] die [realadduser] User ftp already exists
-- RESTART MARKER (PID 20457 at 1301339605)--
Building global cache for cpanel...Done
/scripts/mysqlpasswd: pass is blank
/scripts/mysqlpasswd: pass is blank
/usr/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
[2011-03-28 15:26:06 -0400] warn [securemysql] Unable to determine MySQL data directory from mysqladmin variables. at /usr/local/cpanel/Cpanel/MysqlUtils.pm line 109
	Cpanel::MysqlUtils::getmysqldir() called at /scripts/securemysql line 140
[2011-03-28 15:29:44 -0400] warn [quotacheck] User file '/var/cpanel/users/root' is empty or non-existent. at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 106
	Cpanel::Config::LoadCpUserFile::_load('root', undef, 0) called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 25
	Cpanel::Config::LoadCpUserFile::load('root') called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 252
	Cpanel::Config::LoadCpUserFile::loadcpuserfile('root') called at /scripts/quotacheck line 84
[2011-03-28 15:29:44 -0400] warn [quotacheck] Failed to load cPanel user file for root at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 28
	Cpanel::Config::LoadCpUserFile::load('root') called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 252
	Cpanel::Config::LoadCpUserFile::loadcpuserfile('root') called at /scripts/quotacheck line 84
[2011-03-28 15:31:15 -0400] info [apache_conf_distiller] 'local' datastore in use (/var/cpanel/conf/apache/local)
[2011-03-28 15:31:16 -0400] info [realadduser] No password specified for cpanel
[2011-03-28 15:31:16 -0400] die [realadduser] User cpanel already exists
[2011-03-28 15:31:16 -0400] info [realadduser] No password specified for ftp
[2011-03-28 15:31:16 -0400] die [realadduser] User ftp already exists
-- RESTART MARKER (PID 20208 at 1301340774)--
Building global cache for cpanel...Done
-- RESTART MARKER (PID 20235 at 1301341551)--
Building global cache for cpanel...Done

This is what is the the hostname.err file:

110328 15:45:18  mysqld started
110328 15:45:18 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
InnoDB: ##########################################################
InnoDB:                          WARNING!
InnoDB: The log sequence number in ibdata files is higher
InnoDB: than the log sequence number in the ib_logfiles! Are you sure
InnoDB: you are using the right ib_logfiles to start up the database?
InnoDB: Log sequence number in ib_logfiles is 0 977688, log
InnoDB: sequence numbers stamped to ibdata file headers are between
InnoDB: 589505315 589505315 and 589505315 589505315.
InnoDB: ##########################################################
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110328 15:45:18  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Error: page n:o stored in the page read in is 538976288, should be 3!
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3.
InnoDB: You may have to recover from a backup.
110328 15:45:19  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 
 
 ) ;InnoDB: End of page dump 110328 14:38:24  InnoDB: Page checksum 1508866187, prior-to-4.0.14-form checksum 3161919040 InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105 InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651 InnoDB: Page number (if stored to page already) 538976288, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288 InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 3. InnoDB: You may have to recover from a backup. InnoDB: It is also possible that your operating InnoDB: system has corrupted its own file cache InnoDB: and rebooting your computer removes the InnoDB: error. InnoDB: If the corrupt page is an index page InnoDB: you can also try to fix the corruption InnoDB: by dumping, dropping, and reimporting InnoDB: the corrupt table. You can use CHECK InnoDB: TABLE to scan your table for corruption. InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. InnoDB: Ending processing because of a corrupt database page. se page.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ) ;InnoDB: End of page dump
110328 15:45:19  InnoDB: Page checksum 2747739185, prior-to-4.0.14-form checksum 3161919040
InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105
InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651
InnoDB: Page number (if stored to page already) 538976288,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
110328 15:45:19  mysqld ended

110328 15:46:31  mysqld started
110328 15:46:31 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
InnoDB: ##########################################################
InnoDB:                          WARNING!
InnoDB: The log sequence number in ibdata files is higher
InnoDB: than the log sequence number in the ib_logfiles! Are you sure
InnoDB: you are using the right ib_logfiles to start up the database?
InnoDB: Log sequence number in ib_logfiles is 0 977688, log
InnoDB: sequence numbers stamped to ibdata file headers are between
InnoDB: 589505315 589505315 and 589505315 589505315.
InnoDB: ##########################################################
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110328 15:46:31  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Error: page n:o stored in the page read in is 538976288, should be 3!
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3.
InnoDB: You may have to recover from a backup.
110328 15:46:32  InnoDB: Page dump in ascii and hex (16384 bytes):

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          ) ;InnoDB: End of page dump 110328 14:38:24  InnoDB: Page checksum 1508866187, prior-to-4.0.14-form checksum 3161919040 InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105 InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651 InnoDB: Page number (if stored to page already) 538976288, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288 InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 3. InnoDB: You may have to recover from a backup. InnoDB: It is also possible that your operating InnoDB: system has corrupted its own file cache InnoDB: and rebooting your computer removes the InnoDB: error. InnoDB: If the corrupt page is an index page InnoDB: you can also try to fix the corruption InnoDB: by dumping, dropping, and reimporting InnoDB: the corrupt table. You can use CHECK InnoDB: TABLE to scan your table for corruption. InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. InnoDB: Ending processing because of a corrupt database page. se page.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ) ;InnoDB: End of page dump
110328 15:46:32  InnoDB: Page checksum 2747739185, prior-to-4.0.14-form checksum 3161919040
InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105
InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651
InnoDB: Page number (if stored to page already) 538976288,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
110328 15:46:32  mysqld ended

110328 15:50:57  mysqld started
110328 15:50:57 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
InnoDB: ##########################################################
InnoDB:                          WARNING!
InnoDB: The log sequence number in ibdata files is higher
InnoDB: than the log sequence number in the ib_logfiles! Are you sure
InnoDB: you are using the right ib_logfiles to start up the database?
InnoDB: Log sequence number in ib_logfiles is 0 977688, log
InnoDB: sequence numbers stamped to ibdata file headers are between
InnoDB: 589505315 589505315 and 589505315 589505315.
InnoDB: ##########################################################
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110328 15:50:57  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Error: page n:o stored in the page read in is 538976288, should be 3!
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3.
InnoDB: You may have to recover from a backup.
110328 15:50:57  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 
 ) ;InnoDB: End of page dump 110328 14:38:24  InnoDB: Page checksum 1508866187, prior-to-4.0.14-form checksum 3161919040 InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105 InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651 InnoDB: Page number (if stored to page already) 538976288, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288 InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 3. InnoDB: You may have to recover from a backup. InnoDB: It is also possible that your operating InnoDB: system has corrupted its own file cache InnoDB: and rebooting your computer removes the InnoDB: error. InnoDB: If the corrupt page is an index page InnoDB: you can also try to fix the corruption InnoDB: by dumping, dropping, and reimporting InnoDB: the corrupt table. You can use CHECK InnoDB: TABLE to scan your table for corruption. InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. InnoDB: Ending processing because of a corrupt database page. se page.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ) ;InnoDB: End of page dump
110328 15:50:58  InnoDB: Page checksum 2747739185, prior-to-4.0.14-form checksum 3161919040
InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105
InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651
InnoDB: Page number (if stored to page already) 538976288,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
110328 15:50:58  mysqld ended

[cPanelTristan]

Did you make a full backup of MySQL or do you have a backup of MySQL prior to the password change? Right now, the issue isn't the password change but that InnoDB ib_logfiles don't match the ibdata content, which indicates a crash. It's pretty difficult to fix InnoDB corruption, so restoring from a backup would be preferred.

I am leery of providing any recommendations for InnoDB recovery, since you can majorly mess up your databases if done incorrectly. It would be better to have your datacenter, NOC or provider help with any corruption, or to hire someone to fix it if you do not have sound backups.

Also, always make backups for MySQL especially when using InnoDB before stopping it, restarting it, changing the password, or upgrading it. I've seen too many instances of InnoDB corruption preventing MySQL to restart, and that is exactly what you are showing right now as the issue.

[eglwolf]

I do not have full backups of each of the databases for the various sites on the server. I could check with the NOC to see if they have a backup of the VPS server from the last day and use that to restore. could I take the databases from the /var/lib/mysql directory and backup those and use those after a reinstall?

[cPanelTristan]

You cannot use the existing data in /var/lib/mysql because that data contains corrupted InnoDB data. Please contact your NOC about any VPS backups that exist. You will need the backup to be before the password was changed. You might even want to find out if they have multiple days of backups in case the corruption existed and wasn't apparent until the MySQL restart.

[eglwolf]

Thanks. I am using Wordpress for most of my sites. I use the db-manager plugin for each, and that does a weekly backup of the sites and saves those .sql files. That might be the worse case scenario.

Also I would like to say that once I did the mv /var/lib/mysql to mysql.bak and did a force update of mysql (when I saw all my db where gone) mysql still did not restart. Does that make a difference? Should it have started?

[cPanelTristan]

MySQL would not restart on moving /var/lib/mysql to /var/lib/mysql.bak unless you created /var/lib/mysql with mysql:mysql ownership and ran /scripts/mysqlup --force, since moving the folders with only a restart doesn't recreate MySQL to exist.

[eglwolf]

The NOC does not have any backups. So, can you tell me what I need to do to try and get this working again.

Once I have mysql working again can I use one of the previous databases (one at a time) to see if it has problems which cause mySQL not to start, if so I can remove it and them move to the next one. Maybe I get lucky and fine only one or two that have issues and the others are good?

I can't believe that this all happened from just resetting the password in WHM...

[cPanelTristan]

You can try disabling innodb to begin with and see if it starts up before moving anything. In your /etc/my.cnf file, put the following entry:

skip-innodb

Then try to restart MySQL. If it starts up, then it was definitely InnoDB. Do you know which specific databases were using InnoDB?

[eglwolf]

I did what you asked and still will not restart. my.conf looks like

[mysqld]
skip-innodb
max_connections = 500
thread_cache_size = 4
table_cache = 32K
query_cache_limit=1M
query_cache_size=24M
query_cache_type=1
key_buffer=48M
read_buffer_size=128K
join_buffer_size=128K
connect_timeout=10
max_allowed_packet=1M
max_connect_errors=10
tmp_table_size=128M
max_heap_table_size=128M
log=/var/lib/mysql/general.log
log-slow-queries=/var/lib/mysql/slow.log
long_query_time=5

cPanel Error Logs

[2011-03-28 16:30:01 -0400] warn [mysqluserstore] Unable to query MySQL database at localhost at /usr/local/cpanel/bin/mysqluserstore line 37
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: YES)'
[2011-03-28 16:39:57 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
[2011-03-28 16:39:57 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Access denied for user 'root'@'localhost' (using password: YES) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
 at /usr/local/cpanel/Cpanel/Mysql.pm line 45
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
[2011-03-28 16:41:27 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
[2011-03-28 16:41:27 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
 at /usr/local/cpanel/Cpanel/Mysql.pm line 45
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48

110328 16:39:35  mysqld started
110328 16:39:35 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
110328 16:39:35 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.91-community-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Edition (GPL)
110328 16:41:06 [Note] /usr/sbin/mysqld: Normal shutdown

110328 16:41:14 [Note] /usr/sbin/mysqld: Shutdown complete

110328 16:41:14  mysqld ended

110328 16:41:33  mysqld started
110328 16:41:33 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
110328 16:41:33 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.91-community-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Edition (GPL)
110328 16:41:45 [Note] /usr/sbin/mysqld: Normal shutdown

110328 16:41:55 [Note] /usr/sbin/mysqld: Shutdown complete

110328 16:41:55  mysqld ended

110328 16:41:56  mysqld started
110328 16:41:56 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
110328 16:41:56 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.91-community-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Edition (GPL)

[eglwolf]

OK something buzzer has happened. MySQl started working again... Let me explain what I did. When I first got this VPS I setup the host name to use hostname.net (because I owned the .net version of my domain name). However I forgot that I no longer used that one so I would go into channel and change it to .com. I rebooted the server yesterday and it still came back with .net. and I had not gone to fix it again.

However while waiting for you to reply I went into the VPS control panel and changed it, then I went into cpanel and changed it and verified the new hostname license.

Then I noticed that mysql was working again.... coincidence?

I also noticed that the general.log file within mysql is 2.3 GB. That to me seems large, is that okay or should I do something with that?

[eglwolf]

Another update: in WHM it still shows mysql as down and I keep getting mysql failed @ Mon Mar 28 17:26:13 2011. A restart was attempted automagically.

[cPanelTristan]

Glad that MySQL is now working, although it's strange there were InnoDB errors otherwise on startup attempts unless those were prior errors.

At this point since MySQL itself is running, can you open up a ticket for us to troubleshoot this further? You can open one in WHM > Support Center > Contact cPanel or using the link in my signature. If you have a ticket ID after opening the ticket, please feel free to post it here.

[eglwolf]

Tristan, Thanks for all your help. My ticket request ID 1316277.