Known Issues Status Page - cPanel version 84

[moxie]

Hello Everyone,

Earlier today, there was an update to MariaDB that causes this error to show up in the log files for the database system: InnoDB: Failing assertion: table->can_be_evicted

This error will cause MariaDB to fail upon startup. We've identified this as an upstream issue with MariaDB affecting the following versions of the MariaDB SQL Server:

10.1.42
10.2.28
10.3.19
10.4.9

There is not yet a fix for this issue, although MariaDB is aware of the issue and working on this now.

The only known workaround at this time is to perform a downgrade of the MariaDB RPM packages. While there is not a supported method of downgrading through the cPanel tools, you or your administrator can perform this downgrade using the following command:

yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

If you have any questions about this, please let me know.

This is unfortunate. Thanks for the heads up Lauren.

 

[weetabix]

If we downgrade, will it try to do an update again? Or is it smart enough to wait for next version?

 

[Bashed]

yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel


Total 28 MB/s | 172 MB 00:06
Running rpm_check_debug
Running Transaction Test

Transaction Check Error:
package MariaDB-common-10.2.27-1.el6.x86_64 is already installed
package MariaDB-client-10.2.27-1.el6.x86_64 is already installed
package MariaDB-compat-10.2.27-1.el6.x86_64 is already installed

Error Summary
-------------


Still down.

 

[cPAusaf]

yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel


Total 28 MB/s | 172 MB 00:06
Running rpm_check_debug
Running Transaction Test

Transaction Check Error:
package MariaDB-common-10.2.27-1.el6.x86_64 is already installed
package MariaDB-client-10.2.27-1.el6.x86_64 is already installed
package MariaDB-compat-10.2.27-1.el6.x86_64 is already installed

Error Summary
-------------


Still down.

Hello,

It sounds like there may have been a post scriptlet error which resulted in duplicate packages. May you open a support ticket so we may take a closer look?

Thanks!

 

[bruzli]

woke up with 2 servers in this situation, took some time trying to fix mariadb 10.1 till found this thread.
downgrade fixed it

 

[Hargreaves Standa]

woke up with 2 servers in this situation, took some time trying to fix mariadb 10.1 till found this thread.
downgrade fixed it

Hello,

Can you downgrade if mariadb is already down?

 

[jmginer]

Please, provide a option to migrate from MariaDB to MySQL 8.0 without changing the database names and users.

Thanks!

 

[jacobcolton]

If anyone is interested, we have been given this by cPanel, we haven't tested it yet but I am slightly concerned by the fact this could keep happening until the dodgy package is fixed or updates are disabled:

Changing the RPM Updates preference to "manual" should prevent this from automatically running. A simple one-liner for this would be the following:

sed -i 's/RPMUP=.*/RPMUP=manual/g' /etc/cpupdate.conf ; /scripts/restartsrv_cpsrvd

You can read more about the cpupdate.conf file here:

https://documentation.cpanel.net/display/84Docs/The+cPanel+Update+Configuration+File+-+cpupdate.conf

 

[eva2000]

Hello Everyone,

MariaDB published an update on November 5th, 2019 that can cause MariaDB to fail upon startup on cPanel & WHM servers. We've identified this as an upstream issue with MariaDB (MDEV-19073) affecting the following MariaDB versions:

10.1.42
10.2.28
10.3.19
10.4.9

The following error message is visible in the MariaDB startup output on affected systems:



MariaDB is aware of the issue and working to solve it. We'll update this post with more information on the status of MariaDB case MDEV-19073 as it becomes available.

The only known workaround at this time is to perform a downgrade of the MariaDB RPM packages. While there are no supported methods of downgrading MariaDB using cPanel & WHM, you or your system administrator can manually perform this downgrade using the following command:

yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

Note: The above workaround is intended as a temporary workaround only. If your cPanel & WHM server is configured to automatically update your system packages, the issue will re-occur each time the MariaDB RPMs are upgraded until MariaDB publishes a fix for case MDEV-19073.

If you have any questions about this, please let us know.

Thank you.

From MariaDB Jira bug tracker update with marked fixed in next version = 10.1.43, 10.2.29, 10.3.20, 10.4.10

This is a regression that was caused by the fix of MDEV-20621. MySQL 5.6 (and MariaDB 10.0) introduced eviction of tables from the InnoDB data dictionary cache. Tables that are connected to FOREIGN KEY constraints or FULLTEXT INDEX are exempt of the eviction. With the problematic change, a table that would already be exempt from eviction due to FOREIGN KEY would cause the problem if there also was a FULLTEXT INDEX defined on it.

 

[SigmaWeb]

If anyone is interested, we have been given this by cPanel, we haven't tested it yet but I am slightly concerned by the fact this could keep happening until the dodgy package is fixed or updates are disabled:

Changing the RPM Updates preference to "manual" should prevent this from automatically running. A simple one-liner for this would be the following:

sed -i 's/RPMUP=.*/RPMUP=manual/g' /etc/cpupdate.conf ; /scripts/restartsrv_cpsrvd

You can read more about the cpupdate.conf file here:

https://documentation.cpanel.net/display/84Docs/The+cPanel+Update+Configuration+File+-+cpupdate.conf

In my opinion, cPanel should protect their clients and instead of us having to disable automatic upgrades, cPanel would disable their update servers until they update their packages and issue is fixed.

 

[WebHostPro]

Thanks for the quick fix!

yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

I would disable OS updates first thing before it hits your server. You can turn it back on later and add an exclusion. So far we just had one server effected. Got lucky and caught it quick.

 

[murmaider]

So the downgrade has brought mariadb, which is great, but now some errors which started only after the upgrade still persist after the downgrade for example:

2019-11-06 10:24:43 9 [Warning] InnoDB: Cannot add field `country` in table `rtfcc_db`.`tblclients` because after adding it, the row size is 8690 which is greater than maximum allowed size (8126) for a record on index leaf page.
2019-11-06 10:24:43 9 [Warning] InnoDB: Cannot add field `ns1` in table `rtfcc_db`.`tblhosting` because after adding it, the row size is 8763 which is greater than maximum allowed size (8126) for a record on index leaf page.
2019-11-06 10:24:43 9 [Warning] InnoDB: Cannot add field `configoption4` in table `rtfcc_db`.`tblproducts` because after adding it, the row size is 8744 which is greater than maximum allowed size (8126) for a record on index leaf page.
2019-11-06 10:24:43 9 [Warning] InnoDB: Cannot add field `country` in table `rtfcc_db`.`tblquotes` because after adding it, the row size is 8720 which is greater than maximum allowed size (8126) for a record on index leaf page.
2019-11-06 10:24:43 9 [Warning] InnoDB: Cannot add field `adminunread` in table `rtfcc_db`.`tbltickets` because after adding it, the row size is 8538 which is greater than maximum allowed size (8126) for a record on index leaf page.

Any ideas?, I've read the stuff about the innodb row type being DYNAMIC and that, but it doesn't make sense why these errors never occurred before the upgrade to 10.3.19, and they persist in 10.3.18

 

[niladam]

I'm looking forward in having this fixed ASAP.

 

[LucasRolff]

In my opinion, cPanel should protect their clients and instead of us having to disable automatic upgrades, cPanel would disable their update servers until they update their packages and issue is fixed.

Keep in mind that your server is using the upstream MariaDB repositories, thus it's not something for cPanel to really fix, it's not their update servers that pulled in a broken package, it's the MariaDB repo itself that has decided to not pull the broken package.

cPanel provides information on how to do a rollback, but since the packages are not distributed by them, it's hard for them to control it really.
It's just like if any other package by other repositories got pushed and it happened to be broken.

You as a system administrator should make up with yourself whether you want automated updates enabled for OS packages (and then stuff like this can happen), or if you disable these automated updates and have a maintenance window every X weeks.

 

[kandal]

last night we had the automated upgrade of mariadb in the whm/cpanel which caused us too much pain with our hosted clients !
this is really ridiculous, why there is no notification from cpanel or mariadb, they already knew this could happen ? why it was not stopped before the impact

 

[godzilla]

We are fixing successfully on dozens of servers by running this:

yum downgrade MariaDB-*  -y
whmapi1 update_updateconf RPMUP=manual UPDATES=manual

However I see the issue is FIXED on MariaDB Jira, so I hope the updated packages will be released soon. Maybe the admins have any news about this?

 

[Muhammed Fasal]

We had a bunch of clients complaining about this situation today morning, and we were able to successfully sort and took the precautions to avoid this until a new patch available on MariaDB repos on all those servers. Thanks for providing this workaround, that helped us very much to manage this situation.

But, the same upgrade isn't affected on my own server where a few sites are running (having MariaDB 10.1.42). So I guess this upgrade probably might not affected all the users out there, but majority seems to be affected. Not sure what caused this to happen, and why this weren't affected on some systems. The bug discussed on their official Jira thread here

If anyone require expert assistance to sort this, feel free to contact us.

 

[xigen]

Hi, So MariaDB Server marked the issues as fixed for 10.1.43, 10.2.29, 10.3.20, 10.4.10
(here) about 25 mins ago. I'm I right in saying it now a case of waiting for it to be release here.. Index of /10.3/centos7-amd64/rpms/. ?

 

[Dimiany]

Hello Everyone,

MariaDB published an update on November 5th, 2019 that can cause MariaDB to fail upon startup on cPanel & WHM servers. We've identified this as an upstream issue with MariaDB (MDEV-19073) affecting the following MariaDB versions:

10.1.42
10.2.28
10.3.19
10.4.9

The following error message is visible in the MariaDB startup output on affected systems:



MariaDB is aware of the issue and working to solve it. We'll update this post with more information on the status of MariaDB case MDEV-19073 as it becomes available.

The only known workaround at this time is to perform a downgrade of the MariaDB RPM packages. While there are no supported methods of downgrading MariaDB using cPanel & WHM, you or your system administrator can manually perform this downgrade using the following command:

yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

Note: The above workaround is intended as a temporary workaround only. If your cPanel & WHM server is configured to automatically update your system packages, the issue will re-occur each time the MariaDB RPMs are upgraded until MariaDB publishes a fix for case MDEV-19073.

If you have any questions about this, please let us know.

Thank you.

Hi, my sql version is mysql (10.2.28-MariaDB-log) details-status: down

You say that in the affected platforms this message will exist: " InnoDB: Failing assertion: table->can_be_evicted "

However, the error messages that I occur say :

mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

I cannot locate those files, rather than I have access to my WHM.

My website status is: "Could not connect to MySQL"

Am I also affected from this update? Or is something else that I have to take care ?

Thanks

 

[cPAusaf]

But, the same upgrade isn't affected on my own server where a few sites are running (having MariaDB 10.1.42). So I guess this upgrade probably might not affected all the users out there, but majority seems to be affected. Not sure what caused this to happen, and why this weren't affected on some systems. The bug discussed on their official Jira thread here

One of MariaDB's developers commented the following:

This is a regression that was caused by the fix of MDEV-20621. MySQL 5.6 (and MariaDB 10.0) introduced eviction of tables from the InnoDB data dictionary cache. Tables that are connected to FOREIGN KEY constraints or FULLTEXT INDEX are exempt of the eviction. With the problematic change, a table that would already be exempt from eviction due to FOREIGN KEY would cause the problem if there also was a FULLTEXT INDEX defined on it.

Hi, So MariaDB Server marked the issues as fixed for 10.1.43, 10.2.29, 10.3.20, 10.4.10
(here) about 25 mins ago. I'm I right in saying it now a case of waiting for it to be release here.. Index of /10.3/centos7-amd64/rpms/. ?

That is correct, it looks like they are performing some testing on the fix and building the necessary packages to get pushed out.

Hi, my sql version is mysql (10.2.28-MariaDB-log) details-status: down

You say that in the affected platforms this message will exist: " InnoDB: Failing assertion: table->can_be_evicted "

However, the error messages that I occur say :

mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

I cannot locate those files, rather than I have access to my WHM.

My website status is: "Could not connect to MySQL"

Am I also affected from this update? Or is something else that I have to take care ?

Thanks

It's hard to say without seeing the MySQL error logs (typically /var/lib/mysql/$hostname.err). My recommendation is to open a support ticket so we may verify the issue.

Thanks!