Known Issues Status Page - cPanel version 84

JIKOmetrix

Well-Known Member
Apr 3, 2007
247
49
178
Running the downgrade allow the DB services to come back online for me.

Code:
yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel
 

cPAusaf

Linux Technical Analyst III
Staff member
Aug 24, 2016
41
11
133
Houston, TX
cPanel Access Level
Root Administrator
We have lost access to our WHM so I cannot open any tickets, any advice? please ASAP
Are you able to access the server via SSH? You can get the cPanel support ID with this command:
Code:
/usr/local/cpanel/cpanel -S
SSH access will be needed to investigate the issue. If you are unable to access SSH, then we recommend reaching out to your provider to see what issues may be occurring.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,880
2,258
463
Hello Everyone,

MariaDB has fixed the issue in the following unpublished versions:

10.1.43 [ 23703 ]
10.2.29 [ 23911 ]
10.3.20 [ 23909 ]
10.4.10 [ 23907 ]

We don't have a specific time frame to share on the publication of the updated RPMs, but we'll update this thread as soon as they are published. We strongly recommend avoiding workarounds that take the extra step of disabling automatic updates unless you or your system administration team has a process in-place to monitor the issue and enable automatic updates again once the issue is solved.

Thank you.
 

plague

Well-Known Member
Sep 22, 2006
78
18
158
Brasil
cPanel Access Level
Root Administrator
Twitter
@GOT the problem appears on databases using FULLTEXT KEY indexes. Moving/removing those databases from the data directory enables the server to startup.
You can find this databases looking for file names starting with "FTS*" inside the database directory:

user_database/FTS_000000000005cf3b_CONFIG.ibd
 

cPanelLauren

Product Owner II
Staff member
Nov 14, 2017
13,266
1,300
363
Houston
Update: We've found that MariaDB has removed all affected versions from their repository - At this time the instructions to prevent update using yum-versionlock are no longer necessary.

If you did update and were affected, you would still need to downgrade until the issue is resolved and a new build is pushed out by MariaDB but you will not be upgraded to an affected version.
 

quietFinn

Well-Known Member
Feb 4, 2006
1,850
433
438
Finland
cPanel Access Level
Root Administrator
Update: We've found that MariaDB has removed all affected versions from their repository - At this time the instructions to prevent update using yum-versionlock are no longer necessary.

If you did update and were affected, you would still need to downgrade until the issue is resolved and a new build is pushed out by MariaDB but you will not be upgraded to an affected version.
Had to run yum clean all, then it worked.
 
  • Like
Reactions: cPanelLauren

iero

Member
Jul 2, 2013
10
0
51
cPanel Access Level
Root Administrator
I was so angry in the morning, we have a couple of servers down this morning, we are [Moderator Note: Removed Link To Third-Party URL] and we usually have latest version of whm and litespeed, I had no idea what was wrong until i decided to open a ticket, then i found the issue and we finally resolved downgrading, but it was like 2 hours trying ourselves with our customers calling, not good at all
 
Last edited by a moderator:

kennysamuerto

Member
PartnerNOC
May 20, 2009
24
6
53
cPanel Access Level
Root Administrator
Hello,

In my case, one of my clients was affected by this problem.

The solution in our case, to the little information 24 hours ago when it happened, was to restore a backup of the databases, even though we tried all kinds of alternatives to solve it.

At that point, we decided to remove the databases, and replace them with a backup a few hours earlier.

Therefore, in our case, the bug version is maintained, but everything works correctly after recovering the databases from a backup.

So... What can be expected from the behavior of this server? Should it work well? Should we do downgrade even though everything works correctly?

Thanks
 

webtipster

Well-Known Member
Aug 27, 2016
56
8
58
Atlanta, GA
cPanel Access Level
Website Owner
Praying they have a fix for this soon... Downgrading only solved the problem for some of my servers, might have to roll back more on others. cPanel team please post an update when you know more. Thanks!
 

porkchopzz4

Registered
Nov 6, 2019
3
0
1
Cartersville, Georgia
cPanel Access Level
Root Administrator
Update: We've found that MariaDB has removed all affected versions from their repository - At this time the instructions to prevent update using yum-versionlock are no longer necessary.

If you did update and were affected, you would still need to downgrade until the issue is resolved and a new build is pushed out by MariaDB but you will not be upgraded to an affected version.
Is this a guarantee? I'm scared to turn automatic updates back on and go through this again. I was down 16 hours until support found out what was going on.
 

Leonidas Savvides

Well-Known Member
May 10, 2016
121
5
68
Cyprus
cPanel Access Level
Root Administrator
yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

Note: The above workaround is intended as a temporary workaround only. If your cPanel & WHM server is configured to automatically update your system packages, the issue will re-occur each time the MariaDB RPMs are upgraded until MariaDB publishes a fix for case MDEV-19073.

from this quote
"
If your cPanel & WHM server is configured to automatically update your system packages, the issue will re-occur each time the MariaDB RPMs are upgraded until MariaDB publishes a fix for case MDEV-19073.
"

how to disable server updates from WHM??? is it here:// ??

  • Home »
  • Server Configuration »
  • Update Preferences
if yes choose MANUAL UPDATES ONLY
From
Daily Updates
  1. Automatic
  2. Manual Updates Only
  3. Never Update

????

Or From OS updates

Operating System Package Updates
  1. Automatic
  2. Manual Updates Only
  3. Never Update

disable - choose second option???
this only??

So FINALLY
Set first

Operating System Package Updates

To
Manual Updates Only
And after ssh run downgrade command...?
  • YES CORRECT
  • No
 
Last edited:

Leonidas Savvides

Well-Known Member
May 10, 2016
121
5
68
Cyprus
cPanel Access Level
Root Administrator
I Set first

Operating System Package Updates

To
Manual Updates Only
And after went ssh run the downgrade command...?

I can Not enter SSH I think forgot where is a public key and in what machine (PC)--- I have over 3 years to login since

$ ssh [email protected]

[email protected]: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).


Is it OK - give the command from WHM/Terminal in the browser as shown below????? What other solutions exist???


whm-ssh-terminal.jpg


I mean this cmd here ^^^ >>>
Code:
yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel
 

plague

Well-Known Member
Sep 22, 2006
78
18
158
Brasil
cPanel Access Level
Root Administrator
Twitter
Hello,

In my case, one of my clients was affected by this problem.

The solution in our case, to the little information 24 hours ago when it happened, was to restore a backup of the databases, even though we tried all kinds of alternatives to solve it.

At that point, we decided to remove the databases, and replace them with a backup a few hours earlier.

Therefore, in our case, the bug version is maintained, but everything works correctly after recovering the databases from a backup.

So... What can be expected from the behavior of this server? Should it work well? Should we do downgrade even though everything works correctly?

Thanks
@kennysamuerto downgrade your MariaDB anyway. After the restore if you restart MariaDB, or run a mysqlcheck on databases that use FULLTEXT indexes, it will probably crash again.
I tried the restore before downgrading the installation, and the server crashed in the first restart, after restoring one of those specific databases.
 

cPanelLauren

Product Owner II
Staff member
Nov 14, 2017
13,266
1,300
363
Houston
from this quote
"
If your cPanel & WHM server is configured to automatically update your system packages, the issue will re-occur each time the MariaDB RPMs are upgraded until MariaDB publishes a fix for case MDEV-19073.
"

how to disable server updates from WHM??? is it here:// ??
Again, I'd like to point out the following:

Update: We've found that MariaDB has removed all affected versions from their repository - At this time the instructions to prevent update using yum-versionlock are no longer necessary.

If you did update and were affected, you would still need to downgrade until the issue is resolved and a new build is pushed out by MariaDB but you will not be upgraded to an affected version.

Furthermore, MariaDB have fixed the affected packages and we're just waiting on them to be pushed - we will update here when that is complete, we're hoping for today.
 
  • Like
  • Love
Reactions: Skyla and popeye