Where does CPanel keep MySQL upgrades log files

[albatroz]

I have just upgraded the MySQL version of one of my WHM/Cpanel servers
and would like to review the output of the upgrade procedure as I remember some warnings
recommending to run a fix tables procedure.

 

[Eric]

Howdy,

It should be in /var/lib/mysql/RPM_UPGRADE* there are two files from my update.

Thanks!

 

[gkgcpanel]

I have no such file. Will that appear after the upgrade is done? How about a log while the upgrade is in progress? I ask because I started a MySQL Upgrade via WHM about 35 minutes ago, and it shows to still be running, but the output screen has not moved in over 20 minutes.

# ps ax | grep mysql
9674 pts/1 R+ 0:00 grep mysql
19799 ? S 0:00 whostmgr - mysqlupgrade ./mysqlupgrade
19814 ? Ss 0:00 whostmgr - mysqlupgrade ./mysqlupgrade
19815 ? Ss 0:00 whostmgr - mysqlupgrade ./mysqlupgrade
19903 ? S 0:04 /usr/local/cpanel/scripts/mysqlup
21320 ? S 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/web31.xxxxx.xxx.pid
21376 ? Sl 18:14 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --log-error=/var/lib/mysql/web31.xxxxxx.xxx.err --pid-file=/var/lib/mysql/web31.xxxxxxx.xxx.pid
21408 ? S 0:00 /usr/bin/mysql_upgrade
21907 ? S 0:00 /usr/bin/mysqlcheck --no-defaults --user=root --password=x xxxxxx --user=root --check-upgrade --all-databases --auto-repair --write-binlog

Shows that it is still running...

In the mean time, all my customers on there are complaining of internal server errors, and the log files show:

/usr/lib/libmysqlclient.so.15: version `libmysqlclient_15' not found (required by php)
(I know I have to run an easyapache upgrade afterwards to fix this error), but how can I tell if the process is simply stuck and needs to be kick started again, or should I just wait???

 

[gkgcpanel]

To anyone else having an issue of not knowing if the process is stuck or not. Here's what I found out.

echo "show processlist" | mysql

That showed me that it was doing the following:

7 root localhost xxxxxxxx_xxxxx Query 8947 Repair with keycache REPAIR TABLE `xxxxxxxx_parts`.`NNNN

I then went into /var/lib/mysql/xxxxxxx_parts directory and listed all files in reverse order:

ls -ltrh

Had 2 files for NNNN.MYI and NNNN.TMD the TMD is a temporary file of the .MYI file. The MYI file is 1.6GB and the TMD file is currently at 236MB and growing.

So the MySQL Check is still running and has hit this large table that is currently getting repaired.

 

[cPanelKenneth]

If you are using the MySQL Upgrade Wizard to change your MySQL version, the log will be in /var/cpanel/logs.

If you are doing it manually, via /scripts/mysqlup, there will be no log file.

If the change is merely an update that occurs during upcp (e.g. updating from 5.1.10 to 5.1.12) then the output is logged to /var/cpanel/updatelogs.

The files Eric pointed to you earlier are created by the MySQL RPM. Their existence will depend upon the MySQL version.

 

[gkgcpanel]

Kenneth,

Thank you. That is the information I was looking for.

Now, I seem to have another problem. That same server that was upgraded last week, is now failing with MySQL Dumps when I do a server to server transfer or a /scripts/pkgacct backup.

The error is:

warn [pkgacct] USERNAME_frontier: mysqldump: Couldn't execute 'SHOW FUNCTION STATUS WHERE Db = 'sojodril_frontier'': Column count of mysql.proc is wrong. Expected 20, found 16. Created with MySQL 50095, now running 50525. Please use mysql_upgrade to fix this error. (1558)

I found this link: Please use mysql_upgrade to fix this error. (1558) | Ciuly's Blog - Reinventing the wheel where someone used the mysql_upgrade to fix this error.

Is that safe to use with cPanel? Reading the man page for mysql_upgrade, this should have been called after doing a mysqlup within cPanel, and that It supersedes the older mysql_fix_privilege_tables script, which has been removed in MySQL 5.5.

Did this not get run? And is it safe to run it?

 

[gkgcpanel]

So it looks like I have a catch 22 here... I need to run the upgrade_mysql command to fix errors in several databases.
This forum post: http://forums.cpanel.net/f5/drive-critical-dev-loop0-var-tmp-97-full-190772.html explains the problems I'm seeing "Incorrect key" etc...

So the first step before running any fix programs is to do a mysqldump of all databases. But when it comes to a table that is crashed, the mysqldump abruptly stops and says that the table must be fixed first.

Of course before fixing a table, it is also recommended that you do a mysqldump first. But you can't do that on a crashed table... So I run myisamchk -r username_TABLE.MYI and fix that one table (without a backup). Then I try another mysqldump of all databases. It will usually get past this one that I just fixed, and crash on another one. So far I have started this procedure 9 times, and sometimes it will even crash again on another table I have already repaired at least once.

This makes it very difficult to get a mysqldump of all databases and what I found out now is that users can't backup their databases either (they get errors that the tables are corrupted/crashed and need to be fixed). The /scripts/pkgacct script also fails, as does the Transfer from another server" option.

So I'm stuck. I can't risk running the mysql_upgrade command without a backup, and I can't back up or do anything with MySQL databases from the WHM because of the crashed tables...

 

[cPanelKenneth]

The mysql_upgrade utility is run automatically during the upgrade (and updates) when the MySQL version is 5.1 or newer.

For your current situation you may want to make a physical copy of /var/lib/mysql; fix all the crashed tables; run mysql_upgrade.

Having the physical copy of /var/lib/mysql will ensure you still have something incase data is lost when fixing tables or running the upgrade utility.

 

[gkgcpanel]

Kenneth,

Did that. Finally fixed all the tables and got a complete mysqldump of all databases (took 21 tries).
Also made a backup copy of the /var/lib/mysql directory.

Then ran the mysql_upgrade

# mysql_upgrade -u root -p
Enter password:
Looking for 'mysql' as: mysql
Looking for 'mysqlcheck' as: mysqlcheck
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
Running 'mysqlcheck with default connection arguments
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
FATAL ERROR: Upgrade failed

Fails miserably...

So now what?

The mysql_upgrade utility is run automatically during the upgrade (and updates) when the MySQL version is 5.1 or newer.

For your current situation you may want to make a physical copy of /var/lib/mysql; fix all the crashed tables; run mysql_upgrade.

Having the physical copy of /var/lib/mysql will ensure you still have something incase data is lost when fixing tables or running the upgrade utility.

- - - Updated - - -

Kenneth,

Did that. Finally fixed all the tables and got a complete mysqldump of all databases (took 21 tries).
Also made a backup copy of the /var/lib/mysql directory.

Then ran the mysql_upgrade

# mysql_upgrade -u root -p
Enter password:
Looking for 'mysql' as: mysql
Looking for 'mysqlcheck' as: mysqlcheck
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
Running 'mysqlcheck with default connection arguments
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
FATAL ERROR: Upgrade failed

Fails miserably...

So now what?

The mysql_upgrade utility is run automatically during the upgrade (and updates) when the MySQL version is 5.1 or newer.

For your current situation you may want to make a physical copy of /var/lib/mysql; fix all the crashed tables; run mysql_upgrade.

Having the physical copy of /var/lib/mysql will ensure you still have something incase data is lost when fixing tables or running the upgrade utility.