SOLVED Fresh cPanel unable to update: File conflicts

[Bretas]

Hy, everyone!

The latest CloudLinux + cPanel server in our fleet is experiencing issues during the updates. This server was deployed on July 12th (that is, five days ago) and it's no longer able to update itself since yesterday. Even though cPanel installed MySQL 5.7 by default, it appears that there are still some old things from MySQL 5.6 laying around. Here's the update log:

[2018-07-17 16:37:07 -0300] file /usr/include/mysql/mysql/service_my_snprintf.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/mysql/service_mysql_string.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/mysql/service_thd_alloc.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/mysql/service_thd_wait.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/mysql/service_thread_scheduler.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/mysql/services.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/mysql/thread_pool_priv.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/mysql_com.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/mysql_embed.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/mysql_version.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/mysqld_ername.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/mysqld_error.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/sql_common.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/sql_state.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/sslopt-case.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/sslopt-longopts.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/include/mysql/sslopt-vars.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/lib64/mysql/libmysqlclient.a from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/lib64/mysql/libmysqlservices.a from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/share/man/man1/comp_err.1.gz from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] file /usr/share/man/man1/mysql_config.1.gz from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64
[2018-07-17 16:37:07 -0300] see https://go.cpanel.net/rpmcheckfailed for more information
[2018-07-17 16:37:07 -0300] The Administrator will be notified to review this output when this script completes
=> Log closed Tue Jul 17 16:37:07 2018
[2018-07-17 16:37:07 -0300] 17% complete

However, the package "MySQL56-devel" doesn't seem to be installed at all:

[root@hostname]# yum list installed mysql*   
Plugins carregados: fastestmirror, langpacks, rhnplugin, universal-hooks 
This system is receiving updates from CLN. 
Loading mirror speeds from cached hostfile 
* cpanel-addons-production-feed: 149.56.181.210 
* cloudlinux-x86_64-server-7: xmlrpc.cln.cloudlinux.com 
* epel: epel.mirror.constant.com 
Pacotes instalados 
mysql-community-client.x86_64                                                            5.7.22-1.el7                                                        @mysql57-community 
mysql-community-common.x86_64                                                            5.7.22-1.el7                                                        @mysql57-community 
mysql-community-devel.x86_64                                                             5.7.22-1.el7                                                        @mysql57-community 
mysql-community-libs.x86_64                                                              5.7.22-1.el7                                                        @mysql57-community 
mysql-community-libs-compat.x86_64                                                       5.7.22-1.el7                                                        @mysql57-community 
mysql-community-server.x86_64                                                            5.7.22-1.el7                                                        @mysql57-community 
mysql57-community-release.noarch                                                         el7-11                                                              installed

Here's a comprehensive list of the installed packages containing the term "mysql":

[root@hostname ~]# rpm -qa | grep mysql 
alt-python27-MySQL-mysql5.5-1.2.5-1.el7.cloudlinux.2.x86_64 
alt-php70-mysql5.0-7.0.30-1.el7.x86_64 
alt-php71-mysql5.7-7.1.19-1.el7.x86_64 
alt-php53-mysql5.7-5.3.29-1.el7.x86_64 
alt-php52-mysql5.1-5.2.17-17.el7.x86_64 
alt-php54-mysql5.0-5.4.45-1.el7.x86_64 
alt-php53-mysql-meta-2-3.1.el7.x86_64 
mysql-community-client-5.7.22-1.el7.x86_64 
alt-php70-mysql5.6-7.0.30-1.el7.x86_64 
alt-php71-mysql5.1-7.1.19-1.el7.x86_64 
alt-php56-mysql5.5-5.6.36-1.el7.x86_64 
ea-php56-php-mysqlnd-5.6.36-2.el7.cloudlinux.1.x86_64 
alt-php55-mysql5.7-5.5.38-1.el7.x86_64 
alt-php51-mysql-meta-2-3.1.el7.x86_64 
cpanel-mysql-libs-5.1.73-1.cp1156.x86_64 
alt-php70-mysql5.7-7.0.30-1.el7.x86_64 
alt-php54-mysqlnd-5.4.45-63.el7.x86_64 
alt-php71-mysql-meta-2-3.1.el7.x86_64 
alt-php53-mysqlnd-5.3.29-83.el7.x86_64 
alt-php53-mysql5.1-5.3.29-5.el7.x86_64 
alt-php44-mysql5.7-4.4.9-11.el7.x86_64 
alt-php71-mysql5.6-7.1.19-1.el7.x86_64 
alt-php72-mysql5.7-7.2.7-1.el7.x86_64 
alt-php56-mysql5.7-5.6.36-1.el7.x86_64 
alt-php53-mysql5.0-5.3.29-5.el7.x86_64 
alt-php55-mysql5.5-5.5.38-1.el7.x86_64 
alt-php44-mysql5.6-4.4.9-11.el7.x86_64 
alt-php44-mysql5.0-4.4.9-11.el7.x86_64 
alt-php56-mysql-meta-2-3.1.el7.x86_64 
alt-python27-MySQL-mysql5.0-1.2.5-1.el7.cloudlinux.2.x86_64 
alt-php56-mysql5.6-5.6.36-1.el7.x86_64 
alt-php53-mysql5.6-5.3.29-5.el7.x86_64 
alt-php52-mysql5.5-5.2.17-17.el7.x86_64 
alt-php52-mysql5.0-5.2.17-17.el7.x86_64 
alt-php54-mysql5.7-5.4.45-1.el7.x86_64 
alt-php51-mysql5.0-5.1.6-11.el7.x86_64 
alt-php44-mysql5.1-4.4.9-11.el7.x86_64 
alt-php52-mysql-meta-2-3.1.el7.x86_64 
mysql-community-server-5.7.22-1.el7.x86_64 
alt-php71-mysqlnd-7.1.19-1.el7.x86_64 
alt-php56-mysql5.0-5.6.36-1.el7.x86_64 
alt-php53-mysql5.5-5.3.29-5.el7.x86_64 
alt-php52-mysql5.6-5.2.17-8.el7.x86_64 
alt-php51-mysql5.5-5.1.6-11.el7.x86_64 
mysql57-community-release-el7-11.noarch 
mysql-community-common-5.7.22-1.el7.x86_64 
alt-python27-MySQL-mysql5.6-1.2.5-1.el7.cloudlinux.2.x86_64 
alt-python27-MySQL-mysql5.7-1.2.5-1.el7.cloudlinux.2.x86_64 
cpanel-perl-526-DBD-mysql-4.043-1.cp1170.x86_64 
alt-php70-mysql5.5-7.0.30-1.el7.x86_64 
alt-php72-mysql5.5-7.2.7-1.el7.x86_64 
alt-php52-mysql5.7-5.2.17-1.el7.x86_64 
alt-php54-mysql5.1-5.4.45-1.el7.x86_64 
alt-php55-mysql5.6-5.5.38-1.el7.x86_64 
alt-php51-mysql5.6-5.1.6-11.el7.x86_64 
alt-php51-mysql5.1-5.1.6-11.el7.x86_64 
alt-php54-mysql-meta-2-3.1.el7.x86_64 
alt-php70-mysql-meta-2-3.1.el7.x86_64 
alt-php55-mysqlnd-5.5.38-44.el7.x86_64 
mysql-community-devel-5.7.22-1.el7.x86_64 
alt-php72-mysql5.6-7.2.7-1.el7.x86_64 
alt-php56-mysqlnd-5.6.36-5.el7.x86_64 
alt-php54-mysql5.5-5.4.45-1.el7.x86_64 
alt-php55-mysql5.1-5.5.38-1.el7.x86_64 
alt-php72-mysqlnd-7.2.7-1.el7.x86_64 
alt-php72-mysql5.0-7.2.7-1.el7.x86_64 
alt-php72-mysql-meta-2-3.1.el7.x86_64 
alt-python27-MySQL-mysql5.1-1.2.5-1.el7.cloudlinux.2.x86_64 
alt-php70-mysql5.1-7.0.30-1.el7.x86_64 
alt-php71-mysql5.0-7.1.19-1.el7.x86_64 
alt-php56-mysql5.1-5.6.36-1.el7.x86_64 
alt-php70-mysqlnd-7.0.30-5.el7.x86_64 
alt-php54-mysql5.6-5.4.45-1.el7.x86_64 
alt-php51-mysql5.7-5.1.6-11.el7.x86_64 
alt-php44-mysql5.5-4.4.9-11.el7.x86_64 
alt-php55-mysql-meta-2-3.1.el7.x86_64 
mysql-community-libs-compat-5.7.22-1.el7.x86_64 
alt-php71-mysql5.5-7.1.19-1.el7.x86_64 
alt-php72-mysql5.1-7.2.7-1.el7.x86_64 
cpanel-mysql-5.1.73-1.cp1156.x86_64 
alt-php55-mysql5.0-5.5.38-1.el7.x86_64 
alt-php44-mysql-meta-2-3.1.el7.x86_64 
mysql-community-libs-5.7.22-1.el7.x86_64

[root@hostname ~]# rpm -q --whatprovides /usr/include/mysql/mysql/service_my_snprintf.h 
mysql-community-devel-5.7.22-1.el7.x86_64

SHA1 checksum also confirms those files do belong to MySQL 5.7. This is a fresh server, installed from CludLinux's ISO, nothing special. What's happening here?

Thanks!

 

[cPanelLauren]

HI @Bretas

That is quite odd, I have a couple fresh installed cPanel servers and I'm not experiencing this issue. Did it have MySQL 5.6 installed on it before you installed cPanel? The only way I can think of that this would occur is if it was present prior to the update/install and for some reason it wasn't removed cleanly.

Since the MYSQL56 package isn't present on your server if you look at

/usr/include/mysql/

Are some of the files listed duplicated?

Thanks!

 

[Bretas]

Thanks, Lauren! Glad to see you again.

Very weird indeed. No files in that directory are duplicated (as in containing multiple similar names). I don't see anything atypical in that directory you mentioned:

[root@hostname ~]# ls -lah /usr/include/mysql/
total 656K
drwxr-xr-x  3 root root 4,0K Jul 12 00:04 .
drwxr-xr-x 69 root root  12K Jul 12 15:54 ..
-rw-r--r--  1 root root 3,6K Mar  4 10:40 big_endian.h
-rw-r--r--  1 root root 2,0K Mar  4 10:40 binary_log_types.h
-rw-r--r--  1 root root 2,5K Mar  4 10:40 byte_order_generic.h
-rw-r--r--  1 root root 2,2K Mar  4 10:40 byte_order_generic_x86.h
-rw-r--r--  1 root root 5,1K Mar  4 10:40 decimal.h
-rw-r--r--  1 root root 4,3K Mar  4 10:40 errmsg.h
-rw-r--r--  1 root root 7,4K Mar  4 10:40 keycache.h
-rw-r--r--  1 root root 3,2K Mar  4 10:40 little_endian.h
-rw-r--r--  1 root root  34K Mar  4 10:40 m_ctype.h
-rw-r--r--  1 root root  11K Mar  4 10:40 m_string.h
-rw-r--r--  1 root root 2,3K Mar  4 10:40 my_alloc.h
-rw-r--r--  1 root root 5,4K Mar  4 10:40 my_byteorder.h
-rw-r--r--  1 root root 1,7K Mar  4 10:40 my_command.h
-rw-r--r--  1 root root 5,1K Mar  4 10:40 my_compiler.h
-rw-r--r--  1 root root  919 Mar  4 11:00 my_config.h
-rw-r--r--  1 root root  12K Mar  4 11:00 my_config_x86_64.h
-rw-r--r--  1 root root 9,2K Mar  4 10:40 my_dbug.h
-rw-r--r--  1 root root 3,0K Mar  4 10:40 my_dir.h
-rw-r--r--  1 root root 6,4K Mar  4 10:40 my_getopt.h
-rw-r--r--  1 root root  23K Mar  4 10:40 my_global.h
-rw-r--r--  1 root root 1,5K Mar  4 10:40 my_list.h
drwxr-xr-x  3 root root 4,0K Jul 12 00:04 mysql
-rw-r--r--  1 root root  24K Mar  4 10:40 mysql_com.h
-rw-r--r--  1 root root 1,3K Mar  4 10:40 mysql_com_server.h
-rw-r--r--  1 root root 121K Mar  4 11:00 mysqld_ername.h
-rw-r--r--  1 root root  45K Mar  4 11:00 mysqld_error.h
-rw-r--r--  1 root root 1,2K Mar  4 10:40 mysql_embed.h
-rw-r--r--  1 root root  29K Mar  4 10:40 mysql.h
-rw-r--r--  1 root root 2,2K Mar  4 10:40 mysql_time.h
-rw-r--r--  1 root root 1023 Mar  4 11:00 mysql_version.h
-rw-r--r--  1 root root 3,3K Mar  4 11:00 mysqlx_ername.h
-rw-r--r--  1 root root 2,8K Mar  4 11:00 mysqlx_error.h
-rw-r--r--  1 root root 1,4K Mar  4 11:00 mysqlx_version.h
-rw-r--r--  1 root root  39K Mar  4 10:40 my_sys.h
-rw-r--r--  1 root root 4,8K Mar  4 10:40 my_thread.h
-rw-r--r--  1 root root 2,6K Mar  4 10:40 my_thread_local.h
-rw-r--r--  1 root root 2,8K Mar  4 10:40 my_xml.h
-rw-r--r--  1 root root  18K Mar  4 10:40 plugin_audit.h
-rw-r--r--  1 root root 7,5K Mar  4 10:40 plugin_ftparser.h
-rw-r--r--  1 root root 5,6K Mar  4 10:40 plugin_group_replication.h
-rw-r--r--  1 root root  26K Mar  4 10:40 plugin.h
-rw-r--r--  1 root root 6,2K Mar  4 10:40 plugin_keyring.h
-rw-r--r--  1 root root 1,6K Mar  4 10:40 plugin_validate_password.h
-rw-r--r--  1 root root 7,6K Mar  4 10:40 sql_common.h
-rw-r--r--  1 root root  15K Mar  4 11:00 sql_state.h
-rw-r--r--  1 root root 2,2K Mar  4 10:40 sslopt-case.h
-rw-r--r--  1 root root 2,9K Mar  4 10:40 sslopt-longopts.h
-rw-r--r--  1 root root 2,9K Mar  4 10:40 sslopt-vars.h
-rw-r--r--  1 root root 5,1K Mar  4 10:40 thr_cond.h
-rw-r--r--  1 root root 5,6K Mar  4 10:40 thr_mutex.h
-rw-r--r--  1 root root 6,0K Mar  4 10:40 thr_rwlock.h
-rw-r--r--  1 root root 2,3K Mar  4 10:40 typelib.h

Prior to deploying this cPanel instance, I didn't install any package so there was nothing there that could have sneaked MySQL Server 5.6 in as a dependency.

I spoke with the sysadmin who runs our cPanel post-installation protocol, and he claims it wasn't necessary to upgrade MySQL to version 5.7 this time as it was installed by default. Indeed I just provisioned two test VMs right now, both from CentOS and CloudLinux's ISO, and installed cPanel on them using that one-liner in your documentation (as I always do). Both of these environments came with MySQL 5.7 out of the box and neither displayed any issues during the upgrade, at least not immediately.

Yum's history confirms it's never seen those packages from MySQL 5.6:

[root@hostname ~]# yum history list MySQL56-devel
Plugins carregados: fastestmirror, langpacks, rhnplugin, universal-hooks
This system is receiving updates from CLN.
IDs de transação ou pacote(s) fornecido(s) inválido(s)
Error: Failed history list

Taking the first file of the log output as an example, the SHA1 checksum matches the file provided by MySQL 5.7:

[2018-07-19 01:24:20 -0300] file /usr/include/mysql/mysql/psi/psi.h from install of MySQL56-devel-5.6.39-1.cp1162.x86_64 conflicts with file from package mysql-community-devel-5.7.22-1.el7.x86_64

[root@hostname ~]# sha1sum /usr/include/mysql/mysql/psi/psi.h
72dc93be3296b92ab5d3f8552e2f40c74229fe45  /usr/include/mysql/mysql/psi/psi.h

Not sure if I can post a link here, but if you Google this sum you'll come across a manifest file from an OpenIndiana repo containing MySQL 5.7.

I just ran "yum update", no dependencies issues were found, update went on just fine.

It's almost as if the RPM database is outdated or something (although MySQL 5.6 never even touched the server). Maybe I should run its rebuild script?

Thanks!

 

[cPanelLauren]

Thanks, Lauren! Glad to see you again.

Good to see you too!!

I spoke with the sysadmin who runs our cPanel post-installation protocol, and he claims it wasn't necessary to upgrade MySQL to version 5.7 this time as it was installed by default

He's right, it shouldn't be necessary, it's just so odd that this conflict is present.

Indeed I just provisioned two test VMs right now, both from CentOS and CloudLinux's ISO, and installed cPanel on them using that one-liner in your documentation (as I always do). Both of these environments came with MySQL 5.7 out of the box and neither displayed any issues during the upgrade, at least not immediately.

Which is making this issue all the more curious!

It's almost as if the RPM database is outdated or something (although MySQL 5.6 never even touched the server). Maybe I should run its rebuild script?

Give it a shot, that is a great idea, are there users on this server yet? I am concerned that there could be other issues with the server that we don't know about yet and if at all possible it may be easiest to wipe it and start over, if there are other users then that wouldn't be as simply done.

 

[Bretas]

Thanks, Lauren! Problem solved!

Before I tried to rebuild the RPM database, the solution was found in an answer posted over here last year:

SOLVED - MySQL56 conflicts with file from package MariaDB-server

For whatever reason, the file "/var/cpanel/cpanel.config" in this brand new server contained the line "mysql-version=5.6" even though it came with MySQL 5.7 installed out of the box. I sorted it out by updating the file to match "mysql-version=5.7" and then ran the following command:

/usr/local/cpanel/scripts/update_local_rpm_versions --edit target_settings.MySQL57 installed

(Note: If someone else out there is experiencing the same issue with MariaDB and stumbled upon this topic, please make sure to adjust the command line above according to the release you are effectively running.)

And that was it. I then tried to update cPanel again and it finally finished successfully.

I just checked "/var/cpanel/cpanel.config" in both of those cPanel test environments I deployed yesterday and the value for mysql-version there is correctly set to 5.7 there. I still have no idea why mysql-version=5.6 was defined in our server...

This server has been in production since last Friday 13th (admittedly not the best day to get a new server online, lesson learned ). Hundreds of customers have been migrated to it ever since. Indeed it would have been a hassle to decommission it but, luckily, we haven't seen any other anomaly in it so far. I'll keep an eye on it anyway.

Thanks, Lauren! Have a nice day!

 

[cPanelLauren]

Hi @Bretas

Thank you so much for posting the resolution to the issue here. I will say I just reloaded my OS on one of my test servers July 16th and don't have the same issue:

# grep  mysql-version /var/cpanel/cpanel.config
mysql-version=5.7

Then tried one of my test vms (these are recycled almost weekly):

[root@v72 ~]# grep mysql-version /var/cpanel/cpanel.config
mysql-version=5.7

So I'm not able to replicate it which is good but also makes the issue you had all the more perplexing. Though I am glad you were able to resolve it on your end and thank you again for updating here.


I hope you have a nice day as well!!

Thanks!