We have a weird situation.
Customer's whm/cpanel server crashes due to hardware failure. Can't recover boot partition, so we spin up another vm and restore accounts/system files from latest backup. Turns out the old server was running MariaDB, but we only installed MySQL. After three days of the customer trying to get all their websites working - the newer ones failed, older ones running just fine - they point out that they think we need to install MariaDB.
We also were able to recover the old server's hard drive and mount it on /mnt so we have all the original files, so we have a full backup standing by.
Ran whm > software > mysql/mariadb upgrade and it failed. Output:
We then tried running mysql_upgrade --force at cli. Also didn't work.
The hypothesis is that we have a combination of MySQl & MariaDB files causing the issue. We have a couple of thoughts on how to proceed. Would it make sense to
1. delete the active mysql directory, attempt the upgrade, then restore from the backup directory? or
2. delete the active directory, restore from the backup directory, and attempt the upgrade? or
3. something else?
Customer's whm/cpanel server crashes due to hardware failure. Can't recover boot partition, so we spin up another vm and restore accounts/system files from latest backup. Turns out the old server was running MariaDB, but we only installed MySQL. After three days of the customer trying to get all their websites working - the newer ones failed, older ones running just fine - they point out that they think we need to install MariaDB.
We also were able to recover the old server's hard drive and mount it on /mnt so we have all the original files, so we have a full backup standing by.
Ran whm > software > mysql/mariadb upgrade and it failed. Output:
Code:
Starting process with log file at /var/cpanel/logs/mysql_upgrade.20170817-084333/unattended_upgrade.log
Obtained version information from system.
Beginning “MariaDB 10.0” upgrade...
Obtained version information from system.
Obtained version information from system.
Ensuring the “MariaDB100” repository is available and working.
checkyum version 22.3
Ensuring that the package “MariaDB-client” with version matching “10.0” is available.
Ensuring that the package “MariaDB-common” with version matching “10.0” is available.
Ensuring that the package “MariaDB-devel” with version matching “10.0” is available.
Ensuring that the package “MariaDB-server” with version matching “10.0” is available.
Ensuring that the package “MariaDB-shared” with version matching “10.0” is available.
Ensuring that the package “coreutils” is available.
Ensuring that the package “grep” is available.
Ensuring that the package “perl-DBI” is available.
Ensuring that the package “shadow-utils” is available.
The “MariaDB100” repository is available and working.
Installing pre-known MariaDB dependencies.
Loaded plugins: fastestmirror, universal-hooks
Loading mirror speeds from cached hostfile
* EA4: 208.43.129.162
* cpanel-addons-production-feed: 208.43.129.162
* base: mirror.web-ster.com
* extras: ftp.osuosl.org
* updates: ftp.osuosl.org
Package 2:shadow-utils-4.1.5.1-24.el7.x86_64 already installed and latest version
Package coreutils-8.22-18.el7.x86_64 already installed and latest version
Package perl-DBI-1.627-4.el7.x86_64 already installed and latest version
Package grep-2.20-2.el7.x86_64 already installed and latest version
Nothing to do
Installed pre-known MariaDB dependencies.
Ensuring MariaDB RPMs for version “10.0”.
Verifying that the MariaDB packages can be installed by doing a test install.
checkyum version 22.3
Loaded plugins: fastestmirror, universal-hooks
Loading mirror speeds from cached hostfile
* EA4: 208.43.129.162
* cpanel-addons-production-feed: 208.43.129.162
* base: mirror.web-ster.com
* extras: ftp.osuosl.org
* updates: ftp.osuosl.org
Package MariaDB-server-10.0.32-1.el7.centos.x86_64 already installed and latest version
Package MariaDB-client-10.0.32-1.el7.centos.x86_64 already installed and latest version
Package MariaDB-devel-10.0.32-1.el7.centos.x86_64 already installed and latest version
Package MariaDB-shared-10.0.32-1.el7.centos.x86_64 already installed and latest version
Package MariaDB-common-10.0.32-1.el7.centos.x86_64 already installed and latest version
Nothing to do
Preinstall check passed.
Running: /usr/local/cpanel/scripts/check_cpanel_rpms --targets=MySQL41,MySQL50,MySQL51,MySQL55,MySQL56,MariaDB100,MariaDB101,MariaDB102 --fix
Installing MariaDB from official repository.
Hooks system enabled.
Checking for and running RPM::Versions 'pre' hooks for any RPMs about to be installed
All required 'pre' hooks have been run
Removing incompatible packages.
Loaded plugins: fastestmirror, universal-hooks
Loading mirror speeds from cached hostfile
* EA4: 208.43.129.162
* cpanel-addons-production-feed: 208.43.129.162
* base: mirror.web-ster.com
* extras: ftp.osuosl.org
* updates: ftp.osuosl.org
Package MariaDB-server-10.0.32-1.el7.centos.x86_64 already installed and latest version
Package MariaDB-client-10.0.32-1.el7.centos.x86_64 already installed and latest version
Package MariaDB-devel-10.0.32-1.el7.centos.x86_64 already installed and latest version
Package MariaDB-shared-10.0.32-1.el7.centos.x86_64 already installed and latest version
Package MariaDB-common-10.0.32-1.el7.centos.x86_64 already installed and latest version
Nothing to do
MariaDB installed from official repository.
Building configuration.
Running: /usr/local/cpanel/bin/build_mysql_conf
Waiting for “mysql” to restart ……waiting for “mysql” to initialize ………finished.
Service Status
mysql (/bin/sh /usr/bin/mysqld_safe) is running as mysql with PID 11115 (systemd+/proc check method).
mysql (/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=cp5.ipns.com.err --open-files-limit=10000 --pid-file=cp5.ipns.com.pid) is running as mysql with PID 11242 (systemd+/proc check method).
Startup Log
Aug 17 08:44:37 cp5.ipns.com systemd[1]: Started MySQL Server.
Aug 17 08:44:37 cp5.ipns.com systemd[1]: Starting MySQL Server...
Aug 17 08:44:38 cp5.ipns.com mysqld_safe[11115]: 170817 08:44:38 mysqld_safe Logging to '/var/lib/mysql/cp5.ipns.com.err'.
Aug 17 08:44:38 cp5.ipns.com mysqld_safe[11115]: 170817 08:44:38 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Log Messages
170817 08:44:38 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
170817 08:44:38 mysqld_safe Logging to '/var/lib/mysql/cp5.ipns.com.err'.
170817 08:44:37 mysqld_safe mysqld from pid file /var/lib/mysql/cp5.ipns.com.pid ended
2017-08-17 08:44:37 9166 [Note] /usr/sbin/mysqld: Shutdown complete
2017-08-17 08:44:34 9166 [Note] /usr/sbin/mysqld: Normal shutdown
mysql restarted successfully.
Looking for 'mysql' as: /usr/bin/mysql
Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
This installation of MySQL is already upgraded to 5.6.37, use --force if you still need to run mysql_upgrade
The 'mysql' service passed the check: mysql (/bin/sh /usr/bin/mysqld_safe) is running as mysql with PID 11115 (systemd+/proc check method).
mysql (/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=cp5.ipns.com.err --open-files-limit=10000 --pid-file=cp5.ipns.com.pid) is running as mysql with PID 11242 (systemd+/proc check method).
The 'mysql' service passed the check: mysql (/bin/sh /usr/bin/mysqld_safe) is running as mysql with PID 11115 (systemd+/proc check method).
mysql (/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=cp5.ipns.com.err --open-files-limit=10000 --pid-file=cp5.ipns.com.pid) is running as mysql with PID 11242 (systemd+/proc check method).
Done building configuration.
Hooks system enabled.
Checking for and running RPM::Versions 'post' hooks for any RPMs about to be installed
All required 'post' hooks have been run
Running: /usr/local/cpanel/scripts/check_cpanel_rpms --targets=MySQL41,MySQL50,MySQL51,MySQL55,MySQL56,MariaDB100,MariaDB101,MariaDB102 --fix
Restarting mysql service.
Waiting for “mysql” to restart ……waiting for “mysql” to initialize ………finished.
Service Status
mysql (/bin/sh /usr/bin/mysqld_safe) is running as mysql with PID 11865 (systemd+/proc check method).
mysql (/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=server.tld.err --open-files-limit=10000 --pid-file=server.tld.pid) is running as mysql with PID 11992 (systemd+/proc check method).
Startup Log
Aug 17 08:46:49 cp5.ipns.com systemd[1]: Started MySQL Server.
Aug 17 08:46:49 cp5.ipns.com systemd[1]: Starting MySQL Server...
Aug 17 08:46:49 cp5.ipns.com mysqld_safe[11865]: 170817 08:46:49 mysqld_safe Logging to '/var/lib/mysql/server.tld.err'.
Aug 17 08:46:50 cp5.ipns.com mysqld_safe[11865]: 170817 08:46:50 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Log Messages
170817 08:46:50 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
170817 08:46:49 mysqld_safe Logging to '/var/lib/mysql/server.tld.err'.
170817 08:46:48 mysqld_safe mysqld from pid file /var/lib/mysql/cp5.ipns.com.pid ended
2017-08-17 08:46:48 11242 [Note] /usr/sbin/mysqld: Shutdown complete
2017-08-17 08:46:46 11242 [Note] /usr/sbin/mysqld: Normal shutdown
mysql restarted successfully.
We expected to upgrade to 10.0; however, the system is currently on version 5.6. at /usr/local/cpanel/Whostmgr/Mysql/Upgrade.pm line 839.
Obtained version information from system.
The upgrade process failed!
You should correct any errors listed in the output window and retry the upgrade process beginning at the failed step.The hypothesis is that we have a combination of MySQl & MariaDB files causing the issue. We have a couple of thoughts on how to proceed. Would it make sense to
1. delete the active mysql directory, attempt the upgrade, then restore from the backup directory? or
2. delete the active directory, restore from the backup directory, and attempt the upgrade? or
3. something else?
