MariaDB error after update 10.2.28

plague

Well-Known Member
Sep 22, 2006
67
14
158
Brasil
cPanel Access Level
Root Administrator
Twitter
Google shows this (https://forums.cpanel.net/threads/mariadb-failing-to-start.662601/) link while searching for this erros but it seems like it was deleted, so I'm opening this one that seems to be related.

After upgrading from 10.2.27 to 10.2.28 ( upcp automated update ) MariaDB was not starting up.
I used "innodb_force_recovery = 6" to bring the server back and found that a mysqlcheck crashed the service when checking databases using "FULLTEXT KEY" indexes.
Moving/removing those database's folders from /var/lib/mysql allowed me to restart MariaDB without innodb_force_recovery.
I was than able to recreate those databases and restore a full and working backup, but again when a restart was applyed, MariaDB crashed.

Downgrading to 10.2.27 solved the problem.

Logs was showing this:

Code:
2019-11-05 21:32:10 140612296050880 [Warning] InnoDB: io_setup() failed with EAGAIN. Will make 5 attempts before giving up.
2019-11-05 21:32:10 140612296050880 [Warning] InnoDB: io_setup() attempt 1.
2019-11-05 21:32:10 140612296050880 [Warning] InnoDB: io_setup() attempt 2.
2019-11-05 21:32:11 140612296050880 [Warning] InnoDB: io_setup() attempt 3.
2019-11-05 21:32:11 140612296050880 [Warning] InnoDB: io_setup() attempt 4.
2019-11-05 21:32:12 140612296050880 [Warning] InnoDB: io_setup() attempt 5.
2019-11-05 21:32:12 140612296050880 [ERROR] InnoDB: io_setup() failed with EAGAIN after 5 attempts.
2019-11-05 21:32:12 140612296050880 [Note] InnoDB: You can disable Linux Native AIO by setting innodb_use_native_aio = 0 in my.cnf
2019-11-05 21:32:12 140612296050880 [Warning] InnoDB: Warning: Linux Native AIO disabled because _linux_create_io_ctx() failed. To get rid of this warning you can try increasing system fs.aio-max-nr to 1048576 or larger or setting innodb_use_native_aio = 0 in my.cnf
And after adding "innodb_use_native_aio = 0", it turned to this:

2019-11-06 01:07:25 0x7f100ce4d8c0 InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.28/storage/innobase/dict/dict0dict.cc line 1467
InnoDB: Failing assertion: table->can_be_evicted
 

Valetia

Well-Known Member
Jun 20, 2002
214
10
168
cPanel Access Level
Root Administrator
Indeed, I saw that very thread before it was deleted and it was very helpful, containing the exact command needed to immediately downgrade MariaDB and restore it back to a working state:

Code:
yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel
I am astonished and disappointed that such a useful thread was deleted.

Edit: Also, it would seem a good idea to prevent MariaDB from being automatically updated during this time by editing the file /etc/yum.conf and adding MariaDB* as an additional entry to the exclude= line.
 
Last edited:
Dec 14, 2016
6
0
1
Kenya
cPanel Access Level
Root Administrator
Indeed, I saw that very thread before it was deleted and it was very helpful, containing the exact command needed to immediately downgrade MariaDB and restore it back to a working state:

Code:
yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel
I am astonished and disappointed that such a useful thread was deleted.
Can you downgrade if mariadb is already down?
 

chirpy

Well-Known Member
Verifed Vendor
Jun 15, 2002
13,462
25
473
Go on, have a guess
Code:
yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel
Thank you for reposting that, had this on all of our servers with a mix of MariaDB versions and this resolved it for now.
 
  • Like
Reactions: th3pirat3

Bayern

Registered
Aug 24, 2017
4
0
1
Finland
cPanel Access Level
Root Administrator
Indeed, I saw that very thread before it was deleted and it was very helpful, containing the exact command needed to immediately downgrade MariaDB and restore it back to a working state:

Code:
yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel
I am astonished and disappointed that such a useful thread was deleted.
A BIG thanks to you & plague, I had my MariaDB down this morning & solved it with this!
 

IdleServ

Well-Known Member
Oct 27, 2003
52
2
158
Thanks, the downgrade worked for us too.

What is this bug? Had the failure on 1 server but another server upgraded just fine.
 

Clouseau

Active Member
Jan 17, 2015
29
0
1
cPanel Access Level
Root Administrator
This must be a bug. Our mariadb also woudln't start so dowgrade fixed the problem.

In Progress - MySQL® service failures after update to latest version there is some in Progress report

This is what we have in log file:

2019-11-06 2:30:49 140427190339328 [Note] /usr/sbin/mysqld: Shutdown complete

2019-11-06 2:30:54 140288818653440 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backo
ff, increase buffer pool at least up to 20MB.

2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: The InnoDB memory heap is disabled
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Compressed tables use zlib 1.2.7
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Using Linux native AIO
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Using SSE crc32 instructions
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Completed initialization of buffer pool
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Highest supported file format is Barracuda.
2019-11-06 02:30:54 7f978933a900 InnoDB: Assertion failure in thread 140288818653440 in file dict0dict.cc line 1493
InnoDB: Failing assertion: table->can_be_evicted
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to System Dashboard - Jira
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: MySQL :: MySQL 5.6 Reference Manual :: 14.21.2 Forcing InnoDB Recovery
InnoDB: about forcing recovery.
191106 2:30:54 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see MariaDB Community Bug Reporting

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.42-MariaDB
key_buffer_size=536870912
read_buffer_size=2097152
max_used_connections=0
max_threads=5002
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 21116367 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55961d851a5e]
/usr/sbin/mysqld(handle_fatal_signal+0x305)[0x55961d374455]
sigaction.c:0(__restore_rt)[0x7f9788f2b5f0]
:0(__GI_raise)[0x7f9786fc8337]
:0(__GI_abort)[0x7f9786fc9a28]
/usr/sbin/mysqld(+0x9e3db0)[0x55961d790db0]
/usr/sbin/mysqld(+0x9f656b)[0x55961d7a356b]
/usr/sbin/mysqld(+0x9f6f87)[0x55961d7a3f87]
/usr/sbin/mysqld(+0x9e35e0)[0x55961d7905e0]
/usr/sbin/mysqld(+0xa37078)[0x55961d7e4078]
/usr/sbin/mysqld(+0xa43182)[0x55961d7f0182]
/usr/sbin/mysqld(+0x8b7f83)[0x55961d664f83]
/usr/sbin/mysqld(+0x949ee9)[0x55961d6f6ee9]
/usr/sbin/mysqld(+0x866785)[0x55961d613785]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x64)[0x55961d3767c4]
/usr/sbin/mysqld(+0x44b630)[0x55961d1f8630]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0xa1a)[0x55961d1f9f2a]
/usr/sbin/mysqld(+0x3a1ba1)[0x55961d14eba1]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x550)[0x55961d152a80]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f9786fb4505]
/usr/sbin/mysqld(+0x3990dd)[0x55961d1460dd]
The manual page at MySQL :: MySQL 8.0 Reference Manual :: B.4.3.3 What to Do If MySQL Keeps Crashing contains
information that should help you find out what is causing the crash.
 

uk01

Well-Known Member
Dec 31, 2009
179
18
68
yes! yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel
resolved it for us too. I have spent 3 hours from 6am this morning on this.
It happened at 1.10am after a cpanel update.

I couldn't understand it as there was no corrupt databases, but this downgrade has resolved it thankfully, thank you for who posted this. Much appreciated.
 

uk01

Well-Known Member
Dec 31, 2009
179
18
68
For info, our server was is on mariaDB 10.1.41 (and I guess updated to 10.1.42) so it seems this bug affects 10.1 and 10.2.
 

dexus

Well-Known Member
Jan 14, 2006
176
11
168
cPanel Access Level
Root Administrator
Downgrade is a temporary solution. Next upcp (rpmup) will again update MariaDB, and cause the same issue.

We should probably exclude MariaDB upgrades until we have proper informations and official solution.

@cPanel, can we please get some official note and recommendations regarding this issue?