doveadm was terminated because the system is low on memory

Hi everyone,

Suddenly our mail server stopped working and we keep receiving Cpanel email notifications saying:

Out of memory: The process “doveadm” was terminated because the system is low on memory.
In order to avoid a system crash due to low memory, the kernel terminated the process named “doveadm” with thePID “22260”.

Code:

For addtional details, see the attached dmesg log dump.
[80119.994247] [22673] 0 22673 77061 35855 109 0 0 lmtp
[80119.994386] [22674] 0 22674 62295 29350 89 0 0 lmtp
[80119.994522] [22676] 0 22676 77061 34434 100 0 0 lmtp
[80119.994671] [22677] 0 22677 62724 30888 93 0 0 lmtp
[80119.994875] [22678] 0 22678 77061 33572 102 0 0 lmtp
[80119.995036] [22680] 0 22680 77061 33274 100 0 0 lmtp
[80119.995216] [22681] 0 22681 62460 29904 90 0 0 lmtp
[80119.995384] [22723] 0 22723 11034 745 26 0 0 /usr/local/cpan
[80119.995565] Out of memory: Kill process 22260 (doveadm) score 74 or sacrifice child
[80119.995712] Killed process 22260 (doveadm) total-vm:713376kB, anon-rss:237012kB, file-rss:0kB

Dovecot's logs showing the following:

Code:

Oct 22 11:33:34 vps122722 dovecot: lmtp(swat@example.net): Fatal: block_alloc(268435456): Out of memory

Oct 22 11:33:34 vps122722 dovecot: lmtp(swat@example.net): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x8bd1a) [0x7f7ffca41d1a] -> /usr/lib64/dovecot/libdovecot.so.0(+0x8bd86) [0x7f7ffca41d86] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f7ffc9e1d2d] -> /usr/lib64/dovecot/libdovecot.so.0(+0xa4381) [0x7f7ffca5a381] -> /usr/lib64/dovecot/libdovecot.so.0(+0xa43bc) [0x7f7ffca5a3bc] -> /usr/lib64/dovecot/libdovecot-storage.so.0(maildir_uidlist_refresh+0x392) [0x7f7ffcd34d62] -> /usr/lib64/dovecot/libdovecot-storage.so.0(maildir_uidlist_get_mailbox_guid+0x6a) [0x7f7ffcd35eaa] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x6a9e0) [0x7f7ffcd2d9e0] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x9f367) [0x7f7ffcd62367] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_get_metadata+0x6e) [0x7f7ffcd0737e] -> /usr/lib64/dovecot/libdovecot-lda.so.0(mail_deliver_deduplicate_guid_if_needed+0x80) [0x7f7ffcff88c0] -> /usr/lib64/dovecot/libdovecot-lda.so.0(mail_deliver_save+0x196) [0x7f7ffcff91e6] -> /usr/lib64/dovecot/libdovecot-lda.so.0(mail_deliver+0x11b) [0x7f7ffcff95ab] -> dovecot/lmtp() [0x40681d] -> dovecot/lmtp(cmd_data+0x1ed) [0x406eed] -> dovecot/lmtp(client_input_handle+0x20b) [0x40559b] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x5b) [0x7f7ffca5463b] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xb7) [0x7f7ffca55bf7] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x25) [0x7f7ffca546e5] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f7ffca54888] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f7ffc9e7b33] -> dovecot/lmtp(main+0x169) [0x404e49] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f7ffc640d1d] -> dovecot/lmtp() [0x404c19]

Oct 22 11:33:34 vps122722 dovecot: lmtp(swat@example.net): Fatal: master: service(lmtp): child 24838 returned error 83 (Out of memory (service lmtp { vsz_limit=512 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)

Oct 22 11:33:34 vps122722 dovecot: lmtp(swat@example.net): Fatal: master: service(lmtp): child 24835 returned error 83 (Out of memory (service lmtp { vsz_limit=512 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)

We've tried deleting the mail queue from WHM and restarting Dovecot but in vain...

Can anyone please help?

Thank you!

 

Thanks CpanelNick for your quick response, it seems like increasing the Memory limit doesn't fix the problem, the error evolved to this :

Code:

Fatal: master: service(lmtp): child 30376 returned error 83 (Out of memory (service lmtp { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)

My guess is that it's not a pure memory problem as nothing has changed before and after the bug in term of load and usage.