The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

mySQL Fails after mySQL Password Reset

Discussion in 'General Discussion' started by eglwolf, Mar 28, 2011.

  1. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    185
    Likes Received:
    0
    Trophy Points:
    166
    I reset the password on my mysql (5.01) database through WHM. Now mysql won't restart. Any ideas???
     
    #1 eglwolf, Mar 28, 2011
  2. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,622
    Likes Received:
    24
    Trophy Points:
    238
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    What is the exact error being returned by MySQL when trying to restart it? Also, do you have any entries in /usr/local/cpanel/logs/error_log or in /var/lib/mysql/hostname.err log when you try to restart it? Please replace hostname.err with the server's full hostname for that log file.
     
    #2 cPanelTristan, Mar 28, 2011
  3. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    185
    Likes Received:
    0
    Trophy Points:
    166
    I have tried to force a cpanel update, I have tried to force a mysql update. I also change the reset the root password to the server as well as tried to reset the mysql password.

    One tip I saw was to move and rename /var/lib/mysql to /var/lib/mysql.bak and for an mysqlupdate, then I freaked when all my databases disappeared. So I moved the mysql.bak to mysql


    Here is the error message:
    Starting MySQL. ERROR! Manager of pid-file quit without updating file.

    I just restarted my server and here is what is in the /usr/cpanel/log/error_log file:

    Code:
    [2011-03-28 14:23:12 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
[2011-03-28 14:23:12 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
 at /usr/local/cpanel/Cpanel/Mysql.pm line 45
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
[2011-03-28 14:27:12 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
[2011-03-28 14:27:12 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
 at /usr/local/cpanel/Cpanel/Mysql.pm line 45
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
/usr/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
[2011-03-28 14:27:42 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
[2011-03-28 14:27:42 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
 at /usr/local/cpanel/Cpanel/Mysql.pm line 45
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
[2011-03-28 14:30:01 -0400] warn [mysqluserstore] Unable to query MySQL database at localhost at /usr/local/cpanel/bin/mysqluserstore line 37
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
/usr/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
[2011-03-28 14:38:42 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
[2011-03-28 14:38:42 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
 at /usr/local/cpanel/Cpanel/Mysql.pm line 45
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
-- RESTART MARKER (PID 25793 at 1301339011)--
[2011-03-28 15:03:32 -0400] info [cpanellogd] Unable to load summary files: /var/cpanel/bandwidth/smb.hour, /var/cpanel/bandwidth/smb.5min
Building global cache for cpanel...Done
[2011-03-28 15:09:16 -0400] warn [quotacheck] User file '/var/cpanel/users/root' is empty or non-existent. at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 106
	Cpanel::Config::LoadCpUserFile::_load('root', undef, 0) called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 25
	Cpanel::Config::LoadCpUserFile::load('root') called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 252
	Cpanel::Config::LoadCpUserFile::loadcpuserfile('root') called at /scripts/quotacheck line 84
[2011-03-28 15:09:16 -0400] warn [quotacheck] Failed to load cPanel user file for root at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 28
	Cpanel::Config::LoadCpUserFile::load('root') called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 252
	Cpanel::Config::LoadCpUserFile::loadcpuserfile('root') called at /scripts/quotacheck line 84
[2011-03-28 15:12:08 -0400] warn [securemysql] Unable to determine MySQL data directory from mysqladmin variables. at /usr/local/cpanel/Cpanel/MysqlUtils.pm line 109
	Cpanel::MysqlUtils::getmysqldir() called at /scripts/securemysql line 140
[2011-03-28 15:12:18 -0400] info [apache_conf_distiller] 'local' datastore in use (/var/cpanel/conf/apache/local)
[2011-03-28 15:12:22 -0400] info [realadduser] No password specified for cpanel
[2011-03-28 15:12:22 -0400] die [realadduser] User cpanel already exists
[2011-03-28 15:12:22 -0400] info [realadduser] No password specified for ftp
[2011-03-28 15:12:22 -0400] die [realadduser] User ftp already exists
-- RESTART MARKER (PID 20457 at 1301339605)--
Building global cache for cpanel...Done
/scripts/mysqlpasswd: pass is blank
/scripts/mysqlpasswd: pass is blank
/usr/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists!
[2011-03-28 15:26:06 -0400] warn [securemysql] Unable to determine MySQL data directory from mysqladmin variables. at /usr/local/cpanel/Cpanel/MysqlUtils.pm line 109
	Cpanel::MysqlUtils::getmysqldir() called at /scripts/securemysql line 140
[2011-03-28 15:29:44 -0400] warn [quotacheck] User file '/var/cpanel/users/root' is empty or non-existent. at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 106
	Cpanel::Config::LoadCpUserFile::_load('root', undef, 0) called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 25
	Cpanel::Config::LoadCpUserFile::load('root') called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 252
	Cpanel::Config::LoadCpUserFile::loadcpuserfile('root') called at /scripts/quotacheck line 84
[2011-03-28 15:29:44 -0400] warn [quotacheck] Failed to load cPanel user file for root at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 28
	Cpanel::Config::LoadCpUserFile::load('root') called at /usr/local/cpanel/Cpanel/Config/LoadCpUserFile.pm line 252
	Cpanel::Config::LoadCpUserFile::loadcpuserfile('root') called at /scripts/quotacheck line 84
[2011-03-28 15:31:15 -0400] info [apache_conf_distiller] 'local' datastore in use (/var/cpanel/conf/apache/local)
[2011-03-28 15:31:16 -0400] info [realadduser] No password specified for cpanel
[2011-03-28 15:31:16 -0400] die [realadduser] User cpanel already exists
[2011-03-28 15:31:16 -0400] info [realadduser] No password specified for ftp
[2011-03-28 15:31:16 -0400] die [realadduser] User ftp already exists
-- RESTART MARKER (PID 20208 at 1301340774)--
Building global cache for cpanel...Done
-- RESTART MARKER (PID 20235 at 1301341551)--
Building global cache for cpanel...Done
    This is what is the the hostname.err file:
    Code:
    110328 15:45:18  mysqld started
110328 15:45:18 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
InnoDB: ##########################################################
InnoDB:                          WARNING!
InnoDB: The log sequence number in ibdata files is higher
InnoDB: than the log sequence number in the ib_logfiles! Are you sure
InnoDB: you are using the right ib_logfiles to start up the database?
InnoDB: Log sequence number in ib_logfiles is 0 977688, log
InnoDB: sequence numbers stamped to ibdata file headers are between
InnoDB: 589505315 589505315 and 589505315 589505315.
InnoDB: ##########################################################
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110328 15:45:18  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Error: page n:o stored in the page read in is 538976288, should be 3!
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3.
InnoDB: You may have to recover from a backup.
110328 15:45:19  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 
 
 ) ;InnoDB: End of page dump 110328 14:38:24  InnoDB: Page checksum 1508866187, prior-to-4.0.14-form checksum 3161919040 InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105 InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651 InnoDB: Page number (if stored to page already) 538976288, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288 InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 3. InnoDB: You may have to recover from a backup. InnoDB: It is also possible that your operating InnoDB: system has corrupted its own file cache InnoDB: and rebooting your computer removes the InnoDB: error. InnoDB: If the corrupt page is an index page InnoDB: you can also try to fix the corruption InnoDB: by dumping, dropping, and reimporting InnoDB: the corrupt table. You can use CHECK InnoDB: TABLE to scan your table for corruption. InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. InnoDB: Ending processing because of a corrupt database page. se page.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ) ;InnoDB: End of page dump
110328 15:45:19  InnoDB: Page checksum 2747739185, prior-to-4.0.14-form checksum 3161919040
InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105
InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651
InnoDB: Page number (if stored to page already) 538976288,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
110328 15:45:19  mysqld ended

110328 15:46:31  mysqld started
110328 15:46:31 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
InnoDB: ##########################################################
InnoDB:                          WARNING!
InnoDB: The log sequence number in ibdata files is higher
InnoDB: than the log sequence number in the ib_logfiles! Are you sure
InnoDB: you are using the right ib_logfiles to start up the database?
InnoDB: Log sequence number in ib_logfiles is 0 977688, log
InnoDB: sequence numbers stamped to ibdata file headers are between
InnoDB: 589505315 589505315 and 589505315 589505315.
InnoDB: ##########################################################
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110328 15:46:31  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Error: page n:o stored in the page read in is 538976288, should be 3!
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3.
InnoDB: You may have to recover from a backup.
110328 15:46:32  InnoDB: Page dump in ascii and hex (16384 bytes):

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          ) ;InnoDB: End of page dump 110328 14:38:24  InnoDB: Page checksum 1508866187, prior-to-4.0.14-form checksum 3161919040 InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105 InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651 InnoDB: Page number (if stored to page already) 538976288, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288 InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 3. InnoDB: You may have to recover from a backup. InnoDB: It is also possible that your operating InnoDB: system has corrupted its own file cache InnoDB: and rebooting your computer removes the InnoDB: error. InnoDB: If the corrupt page is an index page InnoDB: you can also try to fix the corruption InnoDB: by dumping, dropping, and reimporting InnoDB: the corrupt table. You can use CHECK InnoDB: TABLE to scan your table for corruption. InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. InnoDB: Ending processing because of a corrupt database page. se page.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ) ;InnoDB: End of page dump
110328 15:46:32  InnoDB: Page checksum 2747739185, prior-to-4.0.14-form checksum 3161919040
InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105
InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651
InnoDB: Page number (if stored to page already) 538976288,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
110328 15:46:32  mysqld ended

110328 15:50:57  mysqld started
110328 15:50:57 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
InnoDB: ##########################################################
InnoDB:                          WARNING!
InnoDB: The log sequence number in ibdata files is higher
InnoDB: than the log sequence number in the ib_logfiles! Are you sure
InnoDB: you are using the right ib_logfiles to start up the database?
InnoDB: Log sequence number in ib_logfiles is 0 977688, log
InnoDB: sequence numbers stamped to ibdata file headers are between
InnoDB: 589505315 589505315 and 589505315 589505315.
InnoDB: ##########################################################
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110328 15:50:57  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Error: page n:o stored in the page read in is 538976288, should be 3!
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3.
InnoDB: You may have to recover from a backup.
110328 15:50:57  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 
 ) ;InnoDB: End of page dump 110328 14:38:24  InnoDB: Page checksum 1508866187, prior-to-4.0.14-form checksum 3161919040 InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105 InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651 InnoDB: Page number (if stored to page already) 538976288, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288 InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 3. InnoDB: You may have to recover from a backup. InnoDB: It is also possible that your operating InnoDB: system has corrupted its own file cache InnoDB: and rebooting your computer removes the InnoDB: error. InnoDB: If the corrupt page is an index page InnoDB: you can also try to fix the corruption InnoDB: by dumping, dropping, and reimporting InnoDB: the corrupt table. You can use CHECK InnoDB: TABLE to scan your table for corruption. InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. InnoDB: Ending processing because of a corrupt database page. se page.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ) ;InnoDB: End of page dump
110328 15:50:58  InnoDB: Page checksum 2747739185, prior-to-4.0.14-form checksum 3161919040
InnoDB: stored checksum 538976288, prior-to-4.0.14-form stored checksum 236985105
InnoDB: Page lsn 538976288 538976288, low 4 bytes of lsn at page end 10651
InnoDB: Page number (if stored to page already) 538976288,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 538976288
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
110328 15:50:58  mysqld ended
     
    #3 eglwolf, Mar 28, 2011
  4. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,622
    Likes Received:
    24
    Trophy Points:
    238
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Did you make a full backup of MySQL or do you have a backup of MySQL prior to the password change? Right now, the issue isn't the password change but that InnoDB ib_logfiles don't match the ibdata content, which indicates a crash. It's pretty difficult to fix InnoDB corruption, so restoring from a backup would be preferred.

    I am leery of providing any recommendations for InnoDB recovery, since you can majorly mess up your databases if done incorrectly. It would be better to have your datacenter, NOC or provider help with any corruption, or to hire someone to fix it if you do not have sound backups.

    Also, always make backups for MySQL especially when using InnoDB before stopping it, restarting it, changing the password, or upgrading it. I've seen too many instances of InnoDB corruption preventing MySQL to restart, and that is exactly what you are showing right now as the issue.
     
    #4 cPanelTristan, Mar 28, 2011
  5. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    185
    Likes Received:
    0
    Trophy Points:
    166
    I do not have full backups of each of the databases for the various sites on the server. I could check with the NOC to see if they have a backup of the VPS server from the last day and use that to restore. could I take the databases from the /var/lib/mysql directory and backup those and use those after a reinstall?
     
    #5 eglwolf, Mar 28, 2011
  6. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,622
    Likes Received:
    24
    Trophy Points:
    238
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    You cannot use the existing data in /var/lib/mysql because that data contains corrupted InnoDB data. Please contact your NOC about any VPS backups that exist. You will need the backup to be before the password was changed. You might even want to find out if they have multiple days of backups in case the corruption existed and wasn't apparent until the MySQL restart.
     
    #6 cPanelTristan, Mar 28, 2011
  7. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    185
    Likes Received:
    0
    Trophy Points:
    166
    Thanks. I am using Wordpress for most of my sites. I use the db-manager plugin for each, and that does a weekly backup of the sites and saves those .sql files. That might be the worse case scenario.

    Also I would like to say that once I did the mv /var/lib/mysql to mysql.bak and did a force update of mysql (when I saw all my db where gone) mysql still did not restart. Does that make a difference? Should it have started?
     
    #7 eglwolf, Mar 28, 2011
  8. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,622
    Likes Received:
    24
    Trophy Points:
    238
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    MySQL would not restart on moving /var/lib/mysql to /var/lib/mysql.bak unless you created /var/lib/mysql with mysql:mysql ownership and ran /scripts/mysqlup --force, since moving the folders with only a restart doesn't recreate MySQL to exist.
     
    #8 cPanelTristan, Mar 28, 2011
  9. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    185
    Likes Received:
    0
    Trophy Points:
    166
    The NOC does not have any backups. So, can you tell me what I need to do to try and get this working again.

    Once I have mysql working again can I use one of the previous databases (one at a time) to see if it has problems which cause mySQL not to start, if so I can remove it and them move to the next one. Maybe I get lucky and fine only one or two that have issues and the others are good?

    I can't believe that this all happened from just resetting the password in WHM...
     
    #9 eglwolf, Mar 28, 2011
  10. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,622
    Likes Received:
    24
    Trophy Points:
    238
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    You can try disabling innodb to begin with and see if it starts up before moving anything. In your /etc/my.cnf file, put the following entry:

    Code:
    skip-innodb
    Then try to restart MySQL. If it starts up, then it was definitely InnoDB. Do you know which specific databases were using InnoDB?
     
    #10 cPanelTristan, Mar 28, 2011
  11. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    185
    Likes Received:
    0
    Trophy Points:
    166
    I did what you asked and still will not restart. my.conf looks like
    Code:
    
[mysqld]
skip-innodb
max_connections = 500
thread_cache_size = 4
table_cache = 32K
query_cache_limit=1M
query_cache_size=24M
query_cache_type=1
key_buffer=48M
read_buffer_size=128K
join_buffer_size=128K
connect_timeout=10
max_allowed_packet=1M
max_connect_errors=10
tmp_table_size=128M
max_heap_table_size=128M
log=/var/lib/mysql/general.log
log-slow-queries=/var/lib/mysql/slow.log
long_query_time=5
    cPanel Error Logs
    Code:
    [2011-03-28 16:30:01 -0400] warn [mysqluserstore] Unable to query MySQL database at localhost at /usr/local/cpanel/bin/mysqluserstore line 37
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: YES)'
[2011-03-28 16:39:57 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
[2011-03-28 16:39:57 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Access denied for user 'root'@'localhost' (using password: YES) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
 at /usr/local/cpanel/Cpanel/Mysql.pm line 45
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
[2011-03-28 16:41:27 -0400] warn [dbstoregrants] Error while connecting to MySQL. Failover enacted. at /usr/local/cpanel/Cpanel/Mysql.pm line 44
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
[2011-03-28 16:41:27 -0400] warn [dbstoregrants] DBI connect('mysql:localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) at /usr/local/cpanel/Cpanel/Mysql.pm line 39
 at /usr/local/cpanel/Cpanel/Mysql.pm line 45
	Cpanel::Mysql::new('Cpanel::Mysql') called at /usr/local/cpanel/bin/dbstoregrants line 74
	main::dump_mysql_sql() called at /usr/local/cpanel/bin/dbstoregrants line 48
    Code:
    110328 16:39:35  mysqld started
110328 16:39:35 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
110328 16:39:35 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.91-community-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Edition (GPL)
110328 16:41:06 [Note] /usr/sbin/mysqld: Normal shutdown

110328 16:41:14 [Note] /usr/sbin/mysqld: Shutdown complete

110328 16:41:14  mysqld ended

110328 16:41:33  mysqld started
110328 16:41:33 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
110328 16:41:33 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.91-community-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Edition (GPL)
110328 16:41:45 [Note] /usr/sbin/mysqld: Normal shutdown

110328 16:41:55 [Note] /usr/sbin/mysqld: Shutdown complete

110328 16:41:55  mysqld ended

110328 16:41:56  mysqld started
110328 16:41:56 [Warning] Changed limits: max_open_files: 65535  max_connections: 500  table_cache: 32512
110328 16:41:56 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.91-community-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Edition (GPL)
     
    #11 eglwolf, Mar 28, 2011
  12. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    185
    Likes Received:
    0
    Trophy Points:
    166
    OK something buzzer has happened. MySQl started working again... Let me explain what I did. When I first got this VPS I setup the host name to use hostname.net (because I owned the .net version of my domain name). However I forgot that I no longer used that one so I would go into channel and change it to .com. I rebooted the server yesterday and it still came back with .net. and I had not gone to fix it again.

    However while waiting for you to reply I went into the VPS control panel and changed it, then I went into cpanel and changed it and verified the new hostname license.

    Then I noticed that mysql was working again.... coincidence?

    I also noticed that the general.log file within mysql is 2.3 GB. That to me seems large, is that okay or should I do something with that?
     
    #12 eglwolf, Mar 28, 2011
  13. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    185
    Likes Received:
    0
    Trophy Points:
    166
    Another update: in WHM it still shows mysql as down and I keep getting mysql failed @ Mon Mar 28 17:26:13 2011. A restart was attempted automagically.
     
    #13 eglwolf, Mar 28, 2011
  14. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,622
    Likes Received:
    24
    Trophy Points:
    238
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Glad that MySQL is now working, although it's strange there were InnoDB errors otherwise on startup attempts unless those were prior errors.

    At this point since MySQL itself is running, can you open up a ticket for us to troubleshoot this further? You can open one in WHM > Support Center > Contact cPanel or using the link in my signature. If you have a ticket ID after opening the ticket, please feel free to post it here.
     
    #14 cPanelTristan, Mar 28, 2011
  15. eglwolf

    eglwolf Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    185
    Likes Received:
    0
    Trophy Points:
    166
    Tristan, Thanks for all your help. My ticket request ID 1316277.
     
    #15 eglwolf, Mar 28, 2011
  16. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,622
    Likes Received:
    24
    Trophy Points:
    238
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Of note, I've replied to the ticket. The issue with MySQL working itself was due to the password change not having properly changed likely due to the fact InnoDB corruption does still exist on that machine (and likely prevented a proper change previously). I had to do the following steps while keeping "skip-innodb" in /etc/my.cnf to get the MySQL root password properly changed:

    1. Stopped MySQL

    Code:
    /etc/init.d/mysql stop
    2. Started MySQL in safe mode with skip-grant-tables

    Code:
    mysqld_safe --skip-grant-tables &
    3. Went into MySQL command line:

    Code:
    mysql
    4. Reset password in MySQL command line for mysql table

    Code:
    mysql> \u mysql 
mysql> update user set password=PASSWORD("passwordhere") where User='root'; 
mysql> flush privileges; 
mysql> quit
    Of note, I've replaced the actual password used from /root/.my.cnf with a phony password. You can find your password by running a cat on /root/.my.cnf file if you don't know what it happens to be:

    Code:
    cat /root/.my.cnf
    5. Stopped MySQL from safe mode and started it

    Code:
    /etc/init.d/mysql stop
/etc/init.d/mysql start
    At that point, mysql command line did work without any further issues. I've kept the skip-innodb option in /etc/my.cnf due to the fact that commenting it out won't let MySQL start still due to the existing InnoDB engine table corruption.
     
    #16 cPanelTristan, Mar 28, 2011
(You must log in or sign up to post here.)
Loading...
Similar Threads - mySQL Fails mySQL
  1. tgv

    set_mysql_password in postwwwacctuser fails

    tgv, Aug 30, 2016, in forum: cPanel Developers
    Replies:
    5
    Views:
    428
    cPanelMichael
    Sep 6, 2016
  2. Solokron

    DNS Only Upgrade Fails - MySQL Dependencies

    Solokron, Mar 23, 2016, in forum: Bind / DNS / Nameserver Issues
    Replies:
    6
    Views:
    629
    cPanelMichael
    Mar 28, 2016
  3. damtech

    MySQL fails to start - quit without updating PID file

    damtech, Jul 29, 2015, in forum: Database Discussions
    Replies:
    7
    Views:
    5,991
    cPanelMichael
    Jul 31, 2015
  4. burn1024

    New Thread Munin mysql2 plugin isn't working

    burn1024, Jun 23, 2017 at 5:41 PM, in forum: General Discussion
    Replies:
    0
    Views:
    26
    burn1024
    Jun 23, 2017 at 5:41 PM
  5. Souther

    MySQL Profiles: user app migration to using remote server

    Souther, Jun 21, 2017 at 11:30 PM, in forum: Database Discussions
    Replies:
    6
    Views:
    63
    Souther
    Jun 22, 2017 at 12:43 PM

Share This Page