doveadm was terminated because the system is low on memory

[soufian]

Hi everyone,

Suddenly our mail server stopped working and we keep receiving Cpanel email notifications saying:

Out of memory: The process “doveadm” was terminated because the system is low on memory.
In order to avoid a system crash due to low memory, the kernel terminated the process named “doveadm” with thePID “22260”.

For addtional details, see the attached dmesg log dump.
[80119.994247] [22673] 0 22673 77061 35855 109 0 0 lmtp
[80119.994386] [22674] 0 22674 62295 29350 89 0 0 lmtp
[80119.994522] [22676] 0 22676 77061 34434 100 0 0 lmtp
[80119.994671] [22677] 0 22677 62724 30888 93 0 0 lmtp
[80119.994875] [22678] 0 22678 77061 33572 102 0 0 lmtp
[80119.995036] [22680] 0 22680 77061 33274 100 0 0 lmtp
[80119.995216] [22681] 0 22681 62460 29904 90 0 0 lmtp
[80119.995384] [22723] 0 22723 11034 745 26 0 0 /usr/local/cpan
[80119.995565] Out of memory: Kill process 22260 (doveadm) score 74 or sacrifice child
[80119.995712] Killed process 22260 (doveadm) total-vm:713376kB, anon-rss:237012kB, file-rss:0kB

Dovecot's logs showing the following:

Oct 22 11:33:34 vps122722 dovecot: lmtp([email protected]): Fatal: block_alloc(268435456): Out of memory

Oct 22 11:33:34 vps122722 dovecot: lmtp([email protected]): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x8bd1a) [0x7f7ffca41d1a] -> /usr/lib64/dovecot/libdovecot.so.0(+0x8bd86) [0x7f7ffca41d86] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f7ffc9e1d2d] -> /usr/lib64/dovecot/libdovecot.so.0(+0xa4381) [0x7f7ffca5a381] -> /usr/lib64/dovecot/libdovecot.so.0(+0xa43bc) [0x7f7ffca5a3bc] -> /usr/lib64/dovecot/libdovecot-storage.so.0(maildir_uidlist_refresh+0x392) [0x7f7ffcd34d62] -> /usr/lib64/dovecot/libdovecot-storage.so.0(maildir_uidlist_get_mailbox_guid+0x6a) [0x7f7ffcd35eaa] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x6a9e0) [0x7f7ffcd2d9e0] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x9f367) [0x7f7ffcd62367] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_get_metadata+0x6e) [0x7f7ffcd0737e] -> /usr/lib64/dovecot/libdovecot-lda.so.0(mail_deliver_deduplicate_guid_if_needed+0x80) [0x7f7ffcff88c0] -> /usr/lib64/dovecot/libdovecot-lda.so.0(mail_deliver_save+0x196) [0x7f7ffcff91e6] -> /usr/lib64/dovecot/libdovecot-lda.so.0(mail_deliver+0x11b) [0x7f7ffcff95ab] -> dovecot/lmtp() [0x40681d] -> dovecot/lmtp(cmd_data+0x1ed) [0x406eed] -> dovecot/lmtp(client_input_handle+0x20b) [0x40559b] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x5b) [0x7f7ffca5463b] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xb7) [0x7f7ffca55bf7] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x25) [0x7f7ffca546e5] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f7ffca54888] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f7ffc9e7b33] -> dovecot/lmtp(main+0x169) [0x404e49] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f7ffc640d1d] -> dovecot/lmtp() [0x404c19]

Oct 22 11:33:34 vps122722 dovecot: lmtp([email protected]): Fatal: master: service(lmtp): child 24838 returned error 83 (Out of memory (service lmtp { vsz_limit=512 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)

Oct 22 11:33:34 vps122722 dovecot: lmtp([email protected]): Fatal: master: service(lmtp): child 24835 returned error 83 (Out of memory (service lmtp { vsz_limit=512 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)

We've tried deleting the mail queue from WHM and restarting Dovecot but in vain...

Can anyone please help?

Thank you!

 

[cPanelNick]

Please try increasing the Process Memory Limit for Mail (MB) under Mailserver Configuration in WHM.

 

[soufian]

Please try increasing the Process Memory Limit for Mail (MB) under Mailserver Configuration in WHM.

Thanks CpanelNick for your quick response, it seems like increasing the Memory limit doesn't fix the problem, the error evolved to this :

Fatal: master: service(lmtp): child 30376 returned error 83 (Out of memory (service lmtp { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)

My guess is that it's not a pure memory problem as nothing has changed before and after the bug in term of load and usage.

 

[SysSachin]

Please try to increase vsz_limit in the exim configartion using below steps :

Login to WHM >> Service Configuration >> Mail Server Configuration.
Once changed we can confirm the modifications by executing the command

doveconf | grep vsz_limit

 

[cPanelMichael]

Fatal: master: service(lmtp): child 30376 returned error 83 (Out of memory (service lmtp { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)

Hello @soufian,

Does the issue persist if you up the value to 2048? If so, would you mind opening a support ticket so that we can take a closer look? You can post the ticket number here so we can update this thread with the outcome.

Thank you.