MariaDB error after update 10.2.28

[plague]

Google shows this (https://forums.cpanel.net/threads/mariadb-failing-to-start.662601/) link while searching for this erros but it seems like it was deleted, so I'm opening this one that seems to be related.

After upgrading from 10.2.27 to 10.2.28 ( upcp automated update ) MariaDB was not starting up.
I used "innodb_force_recovery = 6" to bring the server back and found that a mysqlcheck crashed the service when checking databases using "FULLTEXT KEY" indexes.
Moving/removing those database's folders from /var/lib/mysql allowed me to restart MariaDB without innodb_force_recovery.
I was than able to recreate those databases and restore a full and working backup, but again when a restart was applyed, MariaDB crashed.

Downgrading to 10.2.27 solved the problem.

Logs was showing this:

2019-11-05 21:32:10 140612296050880 [Warning] InnoDB: io_setup() failed with EAGAIN. Will make 5 attempts before giving up.
2019-11-05 21:32:10 140612296050880 [Warning] InnoDB: io_setup() attempt 1.
2019-11-05 21:32:10 140612296050880 [Warning] InnoDB: io_setup() attempt 2.
2019-11-05 21:32:11 140612296050880 [Warning] InnoDB: io_setup() attempt 3.
2019-11-05 21:32:11 140612296050880 [Warning] InnoDB: io_setup() attempt 4.
2019-11-05 21:32:12 140612296050880 [Warning] InnoDB: io_setup() attempt 5.
2019-11-05 21:32:12 140612296050880 [ERROR] InnoDB: io_setup() failed with EAGAIN after 5 attempts.
2019-11-05 21:32:12 140612296050880 [Note] InnoDB: You can disable Linux Native AIO by setting innodb_use_native_aio = 0 in my.cnf
2019-11-05 21:32:12 140612296050880 [Warning] InnoDB: Warning: Linux Native AIO disabled because _linux_create_io_ctx() failed. To get rid of this warning you can try increasing system fs.aio-max-nr to 1048576 or larger or setting innodb_use_native_aio = 0 in my.cnf

And after adding "innodb_use_native_aio = 0", it turned to this:

2019-11-06 01:07:25 0x7f100ce4d8c0 InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.28/storage/innobase/dict/dict0dict.cc line 1467
InnoDB: Failing assertion: table->can_be_evicted

 

[Valetia]

Indeed, I saw that very thread before it was deleted and it was very helpful, containing the exact command needed to immediately downgrade MariaDB and restore it back to a working state:

yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

I am astonished and disappointed that such a useful thread was deleted.

Edit: Also, it would seem a good idea to prevent MariaDB from being automatically updated during this time by editing the file /etc/yum.conf and adding MariaDB* as an additional entry to the exclude= line.

 

[Hargreaves Standa]

Indeed, I saw that very thread before it was deleted and it was very helpful, containing the exact command needed to immediately downgrade MariaDB and restore it back to a working state:

yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

I am astonished and disappointed that such a useful thread was deleted.

Can you downgrade if mariadb is already down?

 

[chirpy]

yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

Thank you for reposting that, had this on all of our servers with a mix of MariaDB versions and this resolved it for now.

 

[Hargreaves Standa]

Thank you for reposting that, had this on all of our servers with a mix of MariaDB versions and this resolved it for now.

Hello,
Can you downgrade if mariadb is already down?

 

[Arturo Leon]

The same happens with MariaDB 10.1.42. Downgrading solves the issue.

 

[Bayern]

Indeed, I saw that very thread before it was deleted and it was very helpful, containing the exact command needed to immediately downgrade MariaDB and restore it back to a working state:

yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

I am astonished and disappointed that such a useful thread was deleted.

A BIG thanks to you & plague, I had my MariaDB down this morning & solved it with this!

 

[JohnBravo]

This resolved it for us well, both on 10.2.x and 10.3.x across systems.

Is there any information on what's causing this, and perhaps more importantly, steps to prevent it in the future?

 

[IdleServ]

Thanks, the downgrade worked for us too.

What is this bug? Had the failure on 1 server but another server upgraded just fine.

 

[James-L]

I also had the above error overnight. Thanks to the command posted by Valetia, the downgrade to MariaDB worked.

 

[Dave_totaal]

I had some servers going down too, the downgrade worked. If i'm not mistaking this is the bug [MDEV-20987] InnoDB fails to start when fts table has FK relation - Jira

 

[Clouseau]

This must be a bug. Our mariadb also woudln't start so dowgrade fixed the problem.

In Progress - MySQL® service failures after update to latest version there is some in Progress report

This is what we have in log file:

2019-11-06 2:30:49 140427190339328 [Note] /usr/sbin/mysqld: Shutdown complete

2019-11-06 2:30:54 140288818653440 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backo
ff, increase buffer pool at least up to 20MB.

2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: The InnoDB memory heap is disabled
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Compressed tables use zlib 1.2.7
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Using Linux native AIO
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Using SSE crc32 instructions
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Completed initialization of buffer pool
2019-11-06 2:30:54 140288818653440 [Note] InnoDB: Highest supported file format is Barracuda.
2019-11-06 02:30:54 7f978933a900 InnoDB: Assertion failure in thread 140288818653440 in file dict0dict.cc line 1493
InnoDB: Failing assertion: table->can_be_evicted
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to System Dashboard - Jira
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: MySQL :: MySQL 5.6 Reference Manual :: 14.21.2 Forcing InnoDB Recovery
InnoDB: about forcing recovery.
191106 2:30:54 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see MariaDB Community Bug Reporting

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.42-MariaDB
key_buffer_size=536870912
read_buffer_size=2097152
max_used_connections=0
max_threads=5002
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 21116367 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55961d851a5e]
/usr/sbin/mysqld(handle_fatal_signal+0x305)[0x55961d374455]
sigaction.c:0(__restore_rt)[0x7f9788f2b5f0]
:0(__GI_raise)[0x7f9786fc8337]
:0(__GI_abort)[0x7f9786fc9a28]
/usr/sbin/mysqld(+0x9e3db0)[0x55961d790db0]
/usr/sbin/mysqld(+0x9f656b)[0x55961d7a356b]
/usr/sbin/mysqld(+0x9f6f87)[0x55961d7a3f87]
/usr/sbin/mysqld(+0x9e35e0)[0x55961d7905e0]
/usr/sbin/mysqld(+0xa37078)[0x55961d7e4078]
/usr/sbin/mysqld(+0xa43182)[0x55961d7f0182]
/usr/sbin/mysqld(+0x8b7f83)[0x55961d664f83]
/usr/sbin/mysqld(+0x949ee9)[0x55961d6f6ee9]
/usr/sbin/mysqld(+0x866785)[0x55961d613785]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x64)[0x55961d3767c4]
/usr/sbin/mysqld(+0x44b630)[0x55961d1f8630]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0xa1a)[0x55961d1f9f2a]
/usr/sbin/mysqld(+0x3a1ba1)[0x55961d14eba1]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x550)[0x55961d152a80]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f9786fb4505]
/usr/sbin/mysqld(+0x3990dd)[0x55961d1460dd]
The manual page at MySQL :: MySQL 8.0 Reference Manual :: B.4.3.3 What to Do If MySQL Keeps Crashing contains
information that should help you find out what is causing the crash.

 

[uk01]

yes! yum downgrade MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel
resolved it for us too. I have spent 3 hours from 6am this morning on this.
It happened at 1.10am after a cpanel update.

I couldn't understand it as there was no corrupt databases, but this downgrade has resolved it thankfully, thank you for who posted this. Much appreciated.

 

[sahostking]

same here - broke a few of our customers cpanel servers and all their sites went down today.

Downgrading Mariadb worked for us too.

 

[uk01]

For info, our server was is on mariaDB 10.1.41 (and I guess updated to 10.1.42) so it seems this bug affects 10.1 and 10.2.

 

[dexus]

Downgrade is a temporary solution. Next upcp (rpmup) will again update MariaDB, and cause the same issue.

We should probably exclude MariaDB upgrades until we have proper informations and official solution.

@cPanel, can we please get some official note and recommendations regarding this issue?

 

[godzilla]

If it can help, I'm temporarily fixing on dozens of server by running

yum downgrade MariaDB-* -y
whmapi1 update_updateconf RPMUP=manual UPDATES=manual

 

[dexus]

For info, our server was is on mariaDB 10.1.41 (and I guess updated to 10.1.42) so it seems this bug affects 10.1 and 10.2.

Yes, exactly, we also had issues on all MariaDB 10.1 servers, for upgrade from 10.1.41 to 10.1.42.

 

[plague]

The bug has been reported here [MDEV-20987] InnoDB fails to start when fts table has FK relation - Jira

I added "MariaDB*" to the "exclude" line on /etc/yum.conf to avoid new updates

 

[infokidunya]

I will same issue but after downgraded MariaDB my SQL run but still on website get error "error in establishing a Database connection" what I can do I don't understand