moxie

Registered
Mar 29, 2017
1
0
0
Huston
cPanel Access Level
Root Administrator
Hello Everyone,

Earlier today, there was an update to MariaDB that causes this error to show up in the log files for the database system: InnoDB: Failing assertion: table->can_be_evicted

This error will cause MariaDB to fail upon startup. We've identified this as an upstream issue with MariaDB affecting the following versions of the MariaDB SQL Server:

10.1.42
10.2.28
10.3.19
10.4.9

There is not yet a fix for this issue, although MariaDB is aware of the issue and working on this now.

The only known workaround at this time is to perform a downgrade of the MariaDB RPM packages. While there is not a supported method of downgrading through the cPanel tools, you or your administrator can perform this downgrade using the following command:

Code:
yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel
If you have any questions about this, please let me know.
This is unfortunate. Thanks for the heads up Lauren.
 

weetabix

Well-Known Member
Oct 26, 2006
62
4
158
If we downgrade, will it try to do an update again? Or is it smart enough to wait for next version?
 

Bashed

Well-Known Member
Dec 18, 2013
128
4
18
cPanel Access Level
Root Administrator
Code:
yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

Total 28 MB/s | 172 MB 00:06
Running rpm_check_debug
Running Transaction Test

Transaction Check Error:
package MariaDB-common-10.2.27-1.el6.x86_64 is already installed
package MariaDB-client-10.2.27-1.el6.x86_64 is already installed
package MariaDB-compat-10.2.27-1.el6.x86_64 is already installed

Error Summary
-------------


Still down.
 

cPAusaf

Linux Technical Analyst III
Staff member
Aug 24, 2016
40
9
83
Houston, TX
cPanel Access Level
Root Administrator
Code:
yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

Total 28 MB/s | 172 MB 00:06
Running rpm_check_debug
Running Transaction Test

Transaction Check Error:
package MariaDB-common-10.2.27-1.el6.x86_64 is already installed
package MariaDB-client-10.2.27-1.el6.x86_64 is already installed
package MariaDB-compat-10.2.27-1.el6.x86_64 is already installed

Error Summary
-------------


Still down.
Hello,

It sounds like there may have been a post scriptlet error which resulted in duplicate packages. May you open a support ticket so we may take a closer look?

Thanks!
 
  • Like
Reactions: cPanelMichael

bruzli

Member
Aug 11, 2006
8
0
151
woke up with 2 servers in this situation, took some time trying to fix mariadb 10.1 till found this thread.
downgrade fixed it
 

jacobcolton

Member
Jan 27, 2005
10
1
153
If anyone is interested, we have been given this by cPanel, we haven't tested it yet but I am slightly concerned by the fact this could keep happening until the dodgy package is fixed or updates are disabled:

Changing the RPM Updates preference to "manual" should prevent this from automatically running. A simple one-liner for this would be the following:

sed -i 's/RPMUP=.*/RPMUP=manual/g' /etc/cpupdate.conf ; /scripts/restartsrv_cpsrvd

You can read more about the cpupdate.conf file here:

https://documentation.cpanel.net/display/84Docs/The+cPanel+Update+Configuration+File+-+cpupdate.conf
 

eva2000

Well-Known Member
Aug 14, 2001
339
16
318
Brisbane, Australia
cPanel Access Level
Root Administrator
Twitter
Hello Everyone,

MariaDB published an update on November 5th, 2019 that can cause MariaDB to fail upon startup on cPanel & WHM servers. We've identified this as an upstream issue with MariaDB (MDEV-19073) affecting the following MariaDB versions:

10.1.42
10.2.28
10.3.19
10.4.9

The following error message is visible in the MariaDB startup output on affected systems:



MariaDB is aware of the issue and working to solve it. We'll update this post with more information on the status of MariaDB case MDEV-19073 as it becomes available.

The only known workaround at this time is to perform a downgrade of the MariaDB RPM packages. While there are no supported methods of downgrading MariaDB using cPanel & WHM, you or your system administrator can manually perform this downgrade using the following command:

yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

Note: The above workaround is intended as a temporary workaround only. If your cPanel & WHM server is configured to automatically update your system packages, the issue will re-occur each time the MariaDB RPMs are upgraded until MariaDB publishes a fix for case MDEV-19073.

If you have any questions about this, please let us know.

Thank you.
From MariaDB Jira bug tracker update with marked fixed in next version = 10.1.43, 10.2.29, 10.3.20, 10.4.10
This is a regression that was caused by the fix of MDEV-20621. MySQL 5.6 (and MariaDB 10.0) introduced eviction of tables from the InnoDB data dictionary cache. Tables that are connected to FOREIGN KEY constraints or FULLTEXT INDEX are exempt of the eviction. With the problematic change, a table that would already be exempt from eviction due to FOREIGN KEY would cause the problem if there also was a FULLTEXT INDEX defined on it.
 

SigmaWeb

Active Member
PartnerNOC
Sep 26, 2006
34
3
158
Athens - Greece
cPanel Access Level
Root Administrator
If anyone is interested, we have been given this by cPanel, we haven't tested it yet but I am slightly concerned by the fact this could keep happening until the dodgy package is fixed or updates are disabled:

Changing the RPM Updates preference to "manual" should prevent this from automatically running. A simple one-liner for this would be the following:

sed -i 's/RPMUP=.*/RPMUP=manual/g' /etc/cpupdate.conf ; /scripts/restartsrv_cpsrvd

You can read more about the cpupdate.conf file here:

https://documentation.cpanel.net/display/84Docs/The+cPanel+Update+Configuration+File+-+cpupdate.conf
In my opinion, cPanel should protect their clients and instead of us having to disable automatic upgrades, cPanel would disable their update servers until they update their packages and issue is fixed.
 

WebHostPro

Well-Known Member
PartnerNOC
Jul 28, 2002
1,700
24
318
LA, Costa RIca
cPanel Access Level
Root Administrator
Twitter
Thanks for the quick fix!

Code:
yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel
I would disable OS updates first thing before it hits your server. You can turn it back on later and add an exclusion. So far we just had one server effected. Got lucky and caught it quick.
 

murmaider

Registered
PartnerNOC
Apr 5, 2012
4
0
51
cPanel Access Level
DataCenter Provider
So the downgrade has brought mariadb, which is great, but now some errors which started only after the upgrade still persist after the downgrade for example:

Code:
2019-11-06 10:24:43 9 [Warning] InnoDB: Cannot add field `country` in table `rtfcc_db`.`tblclients` because after adding it, the row size is 8690 which is greater than maximum allowed size (8126) for a record on index leaf page.
2019-11-06 10:24:43 9 [Warning] InnoDB: Cannot add field `ns1` in table `rtfcc_db`.`tblhosting` because after adding it, the row size is 8763 which is greater than maximum allowed size (8126) for a record on index leaf page.
2019-11-06 10:24:43 9 [Warning] InnoDB: Cannot add field `configoption4` in table `rtfcc_db`.`tblproducts` because after adding it, the row size is 8744 which is greater than maximum allowed size (8126) for a record on index leaf page.
2019-11-06 10:24:43 9 [Warning] InnoDB: Cannot add field `country` in table `rtfcc_db`.`tblquotes` because after adding it, the row size is 8720 which is greater than maximum allowed size (8126) for a record on index leaf page.
2019-11-06 10:24:43 9 [Warning] InnoDB: Cannot add field `adminunread` in table `rtfcc_db`.`tbltickets` because after adding it, the row size is 8538 which is greater than maximum allowed size (8126) for a record on index leaf page.

Any ideas?, I've read the stuff about the innodb row type being DYNAMIC and that, but it doesn't make sense why these errors never occurred before the upgrade to 10.3.19, and they persist in 10.3.18
 

LucasRolff

Well-Known Member
May 27, 2013
129
76
28
cPanel Access Level
Root Administrator
In my opinion, cPanel should protect their clients and instead of us having to disable automatic upgrades, cPanel would disable their update servers until they update their packages and issue is fixed.
Keep in mind that your server is using the upstream MariaDB repositories, thus it's not something for cPanel to really fix, it's not their update servers that pulled in a broken package, it's the MariaDB repo itself that has decided to not pull the broken package.

cPanel provides information on how to do a rollback, but since the packages are not distributed by them, it's hard for them to control it really.
It's just like if any other package by other repositories got pushed and it happened to be broken.

You as a system administrator should make up with yourself whether you want automated updates enabled for OS packages (and then stuff like this can happen), or if you disable these automated updates and have a maintenance window every X weeks.
 
  • Like
Reactions: cPanelLauren

kandal

Registered
Nov 6, 2019
1
0
0
UAE
cPanel Access Level
Root Administrator
last night we had the automated upgrade of mariadb in the whm/cpanel which caused us too much pain with our hosted clients !
this is really ridiculous, why there is no notification from cpanel or mariadb, they already knew this could happen ? why it was not stopped before the impact
 

Muhammed Fasal

Well-Known Member
Aug 9, 2017
54
9
8
India
cPanel Access Level
Root Administrator
We had a bunch of clients complaining about this situation today morning, and we were able to successfully sort and took the precautions to avoid this until a new patch available on MariaDB repos on all those servers. Thanks for providing this workaround, that helped us very much to manage this situation. :)

But, the same upgrade isn't affected on my own server where a few sites are running (having MariaDB 10.1.42). So I guess this upgrade probably might not affected all the users out there, but majority seems to be affected. Not sure what caused this to happen, and why this weren't affected on some systems. The bug discussed on their official Jira thread here

If anyone require expert assistance to sort this, feel free to contact us.
 

Dimiany

Registered
Nov 6, 2019
2
0
1
Thessaloniki
cPanel Access Level
Website Owner
Hello Everyone,

MariaDB published an update on November 5th, 2019 that can cause MariaDB to fail upon startup on cPanel & WHM servers. We've identified this as an upstream issue with MariaDB (MDEV-19073) affecting the following MariaDB versions:

10.1.42
10.2.28
10.3.19
10.4.9

The following error message is visible in the MariaDB startup output on affected systems:



MariaDB is aware of the issue and working to solve it. We'll update this post with more information on the status of MariaDB case MDEV-19073 as it becomes available.

The only known workaround at this time is to perform a downgrade of the MariaDB RPM packages. While there are no supported methods of downgrading MariaDB using cPanel & WHM, you or your system administrator can manually perform this downgrade using the following command:

yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

Note: The above workaround is intended as a temporary workaround only. If your cPanel & WHM server is configured to automatically update your system packages, the issue will re-occur each time the MariaDB RPMs are upgraded until MariaDB publishes a fix for case MDEV-19073.

If you have any questions about this, please let us know.

Thank you.

Hi, my sql version is mysql (10.2.28-MariaDB-log) details-status: down

You say that in the affected platforms this message will exist: " InnoDB: Failing assertion: table->can_be_evicted "

However, the error messages that I occur say :
mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

I cannot locate those files, rather than I have access to my WHM.

My website status is: "Could not connect to MySQL"

Am I also affected from this update? Or is something else that I have to take care ?

Thanks
 

cPAusaf

Linux Technical Analyst III
Staff member
Aug 24, 2016
40
9
83
Houston, TX
cPanel Access Level
Root Administrator
But, the same upgrade isn't affected on my own server where a few sites are running (having MariaDB 10.1.42). So I guess this upgrade probably might not affected all the users out there, but majority seems to be affected. Not sure what caused this to happen, and why this weren't affected on some systems. The bug discussed on their official Jira thread here
One of MariaDB's developers commented the following:
This is a regression that was caused by the fix of MDEV-20621. MySQL 5.6 (and MariaDB 10.0) introduced eviction of tables from the InnoDB data dictionary cache. Tables that are connected to FOREIGN KEY constraints or FULLTEXT INDEX are exempt of the eviction. With the problematic change, a table that would already be exempt from eviction due to FOREIGN KEY would cause the problem if there also was a FULLTEXT INDEX defined on it.
Hi, So MariaDB Server marked the issues as fixed for 10.1.43, 10.2.29, 10.3.20, 10.4.10
(here) about 25 mins ago. I'm I right in saying it now a case of waiting for it to be release here.. Index of /10.3/centos7-amd64/rpms/. ?
That is correct, it looks like they are performing some testing on the fix and building the necessary packages to get pushed out.
Hi, my sql version is mysql (10.2.28-MariaDB-log) details-status: down

You say that in the affected platforms this message will exist: " InnoDB: Failing assertion: table->can_be_evicted "

However, the error messages that I occur say :
mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

I cannot locate those files, rather than I have access to my WHM.

My website status is: "Could not connect to MySQL"

Am I also affected from this update? Or is something else that I have to take care ?

Thanks
It's hard to say without seeing the MySQL error logs (typically /var/lib/mysql/$hostname.err). My recommendation is to open a support ticket so we may verify the issue.

Thanks!