Mailboxes corrupting (dovecot.index file)

[camble]

Hello,
I have recently had a few of my clients saying that their email accounts aren't working and when I've investigated by logging onto Webmail (Roundcube) it shows the following warning/error: "Server Error (OK)".

I paid server admins to look into the issue and for one instance they said that the connection hadn't been closed properly by Outlook Express for a client, however I'm pretty sure that some of the other clients it's happened to are not using Outlook Express so I don't think it's just cause by that program.


The way I've been fixing the corrupt mailboxes is to delete/rename the dovecot.index file for their email account, however I'd like to know how I can prevent this from happening.


Does anybody have any ideas?


Kind regards
Liam

 

[cPanelPeter]

Hello,

Are you seeing any errors in the /var/log/maillog file?

 

[camble]

Hi Peter,
Here is the message (with email addresses changed to prevent spam) that I got from my server admins:

Feb 11 13:04:20 ip-1**-2*5-33-59 dovecot: imap(kevin.s****@loc8********gents.co.uk): Error: broken sync positions in index file /home/loc8/mail/loc8**********nts.co.uk/kevin.s****/dovecot.index
Feb 11 13:04:20 ip-1**-2*5-33-59 dovecot: imap(kevin.s****@loc8********gents.co.uk): Warning: fscking index file /home/loc8/mail/loc8********ents.co.uk/kevin.s****/dovecot.index

 

[domeneas]

Hi,

this is happening to me too. It was one incident 6 months or so ago. Then a couple here and there, but now I have had atleast 10 within the last few weeks telling me it has something to to with load or activity. Seems only to happen to very active users.

I started looking through the logs and found this beauty of someone whom I just deleted the dovecot*.* files from their mail dir to get it working again:

Apr  3 22:21:17 tor dovecot: pop3-login: Login: user=<ellingsen@*****.com>, method=PLAIN, rip=*.*.*.205, lip=*.*.*.207, mpid=23929, TLS, session=<zJjDJCn2lwAClhLN>
Apr  3 22:21:19 tor dovecot: pop3(ellingsen@*****.com): Error: broken sync positions in index file /home/brensel/mail/*****.com/ellingsen/dovecot.index
Apr  3 22:21:19 tor dovecot: pop3(ellingsen@*****.com): Warning: fscking index file /home/brensel/mail/*****.com/ellingsen/dovecot.index
Apr  3 22:21:19 tor dovecot: pop3(ellingsen@*****.com): Error: Fixed index file /home/brensel/mail/*****.com/ellingsen/dovecot.index: log_file_tail_offset 176 -> 144
Apr  3 22:21:19 tor dovecot: pop3(ellingsen@*****.com): Panic: file mail-transaction-log.c: line 350 (mail_transaction_log_set_mailbox_sync_pos): assertion failed: (file_offset >= log->head->saved_tail_offset)
Apr  3 22:21:19 tor dovecot: pop3(ellingsen@*****.com): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0 [0x2b73fe993a7a] -> /usr/lib64/dovecot/libdovecot.so.0 [0x2b73fe993ae6] -> /usr/lib64/dovecot/libdovecot.so.0 [0x2b73fe993473] -> /usr/lib64/dovecot/libdovecot-storage.so.0 [0x2b73fe6ea81b] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_sync_commit+0x89) [0x2b73fe6e2069] -> /usr/lib64/dovecot/libdovecot-storage.so.0 [0x2b73fe66e20d] -> /usr/lib64/dovecot/libdovecot-storage.so.0 [0x2b73fe66d717] -> /usr/lib64/dovecot/libdovecot-storage.so.0 [0x2b73fe66da58] -> /usr/lib64/dovecot/libdovecot-storage.so.0(maildir_storage_sync_init+0xbb) [0x2b73fe66dbfb] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x31) [0x2b73fe69a391] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync+0x27) [0x2b73fe69cae7] -> dovecot/pop3(client_create+0x332) [0x405722] -> dovecot/pop3 [0x40414b] -> dovecot/pop3 [0x404708] -> /usr/lib64/dovecot/libdovecot.so.0 [0x2b73fe9512f2] -> /usr/lib64/d
Apr  3 22:21:19 tor dovecot: pop3(ellingsen@*****.com): Fatal: master: service(pop3): child 23929 killed with signal 6 (core dumps disabled)

 

[cPanelMichael]

It looks like the dovecot.index file for that email account might be corrupted. Please try moving it out of the way to see if that resolves the issue. EX:

mv /home/username/mail/domain.com/email-account/dovecot.index /root/

The file should automatically regenerate upon logging in. I know you mentioned this issue continues to reoccur, so feel free to open a support ticket if this keeps happening so we can take a closer look. You can post the ticket number here so we can update this thread with the outcome.

Thank you.

 

[domeneas]

Yes, that works every time, but the issue keep recurring and while the customer waits for it to be resolved by us his email service is unusable.

I will open a support case.

 

[vestigium]

Yes, that works every time, but the issue keep recurring and while the customer waits for it to be resolved by us his email service is unusable.

I will open a support case.

Hi! I have same problem with my clients.
I already tried to clean dovecot files, but doesn't works.
If you found a solution, can you update this topic?

Thanks!

 

[vestigium]

I solved my problem. In my case occurred when trying to delete any email.
I was deleting the dovecot files from wrong folder (excluding the root folder).
In my error log specified the directory that the problem was occurring:

Apr 14 12:49:13 vps dovecot: imap([email protected]): Warning: fscking index file /home/domain/mail/domain.com/account/.Trash/dovecot.index

Only took into this directory and remove the dovecot files.

 

[cPanelMichael]

I will open a support case.

Feel free to post the ticket number here so we can update this thread with the outcome.

Thank you.

 

[domeneas]

Support Request Id 4879553 is currently open and being investigated

 

[cPanelMichael]

To update, the customer was advised to increase the "Spare Authentication Processes" value for Dovecot to see if that makes a difference in the corruption of the dovecot.index files.

Thank you.

 

[domeneas]

To update, the customer was advised to increase the "Spare Authentication Processes" value for Dovecot to see if that makes a difference in the corruption of the dovecot.index files.

That didn't help, as I still got errors later that day, about 30 minutes after changing the number of available authentication daemons, but oddly do not have any the last 2 days. I have a little hope that maybe the last errors really were the last ones and some log processing time or the likes accounts for that 30-45 min gap.

Cpanel now has core dumps from the server when the error occurred, so hopefully that will tell them something.

 

[irefuse]

That didn't help, as I still got errors later that day, about 30 minutes after changing the number of available authentication daemons, but oddly do not have any the last 2 days. I have a little hope that maybe the last errors really were the last ones and some log processing time or the likes accounts for that 30-45 min gap.

Cpanel now has core dumps from the server when the error occurred, so hopefully that will tell them something.

Has there been any update on your ticket? I am having this exact same issue. larger/more frequently used mailboxes are having the index files become corrupt. At least 1-2 times a month

 

[cPanelMichael]

Has there been any update on your ticket? I am having this exact same issue. larger/more frequently used mailboxes are having the index files become corrupt. At least 1-2 times a month

The customer has not reported additional problems since updating their kernel.

Thank you.

 

[domeneas]

Sorry, I forgot about this thread.

Unfortunately I am not able to update the Kernel, so I cannot test that. I would have to migrate the whole VPS to a new platform, and I'm just not willing to do that with as many users as are on the box in question.

I am seeing improvements from just tweaking the authentication daemons as stated above though. At least I assume it must be from that. The issue is much rarer again now.

 

[camble]

I've not visited this thread for a while but I've consistently had the same problem time and time again, however it does seem to be limited to only a few of my clients (which I know use the email a lot).

I noticed that it was suggested to increase the "Spare Authentication Processes" value - my default value is 2, so what would be an appropriate value to increase to?


Kind regards
Liam

 

[cPanelMichael]

I noticed that it was suggested to increase the "Spare Authentication Processes" value - my default value is 2, so what would be an appropriate value to increase to?

You could try 3 or 4 to see if that makes a difference with this issue.

Thank you.

 

[triantech]

As far as i have seen, deleting the corrupted index file is the only solution. It would be better to
run a cron to remove the file once in a while ( since its automatically regenerated )

 

[camble]

You could try 3 or 4 to see if that makes a difference with this issue.

Thank you.

A couple of weeks ago I changed it to 5 after speaking to a rep from GoDaddy who said 5 would be ok.
Fingers crossed nothing has corrupted since but that doesn't necessarily mean it's solved it as it happened every so often anyway.

The client who it was mainly happening to has gone elsewhere because of the issue so it might not be so easy to know if it does happen.


Kind regards
Liam

 

[camble]

// Update //

Since changing the Spare Authentication Processes to 5, it doesn't seem to have happened "as" often, although it's hard to gauge. However in the last few weeks it seems to have started happening a lot more regularly.

I am having my developer write a script that will allow my clients to go to a url which will remove/rename all dovecot.index files within their account that should fix the issue, although this is obviously not ideal because I'd rather it didn't happen in the first place.


When I looked at the log files previously and asked my server admin to overview them, they said that it was because the connection wasn't closed properly by the email client. My hypothesis is that it's when a user is running a program such as Outlook and doesn't close the program before shutting down their machine, and Outlook doesn't close properly (Not Responding) and they force close it. My reason behind this idea is because I always shut my Outlook down before shutting down and it's never once happened to me, however I know that several of my clients will not do this and will just shut down the machine with programs running.