SOLVED [CPANEL-28981] Rare error when converting from MDBOX back to MAILDIR

[Denis Gomes Franco]

Hey everyone. I think I found a rather unusual, and possibly quite rare, bug.

For quite some time I've been using the MDBOX format due to its efficiency. But I had to convert some accounts back to MAILDIR format.

And then I found out - the hard way, after a customer's complaint - that while the conversion from MAILDIR to MDBOX was flawless, the opposite has a weird bug: all mails stored in folders that had accented characters (eg. FINALIZAÇÃO) were lost.

Thing is, during conversion Cpanel didn't show any warnings. Those folders simply weren't converted and now they are gone. Below I'll summarize the steps to reproduce this issue:
1. Create a CPANEL account using the MDBOX format
2. Create an e-mail account
3. Open that e-mail account in Roundcube
4. Create two folders, one named FINALIZACAO (without accents) and another named FINALIZAÇÃO (with accents)
5. Save some messages to both these folders
6. Run the conversion from MDBOX to MAILDIR
7. Fire up Roundcube again
8. Notice there is only the folder named FINALIZACAO.

Here is the log of the conversion process I ran in order to test this bug:

Starting “RESTORE” for “Account” “domaintest”.
The original files will be removed after the conversion.
Temporarily lifting quota restrictions for “domaintest”.
Changing directory to “/home/domaintest/mail”.
Target “/home/domaintest/mail” on host “cp001.sitesexclusivos.com.br” has 99.16 GB free and requires at least 0 bytes free, which includes space for temporary files.
Creating conversion lock file for “[email protected]”
Cleaning up “/home/domaintest/mail” for “[email protected]”.
Syncing while existing sessions for “[email protected]” are using mdbox (pass 1).
Syncing “[email protected]” mdbox -> maildir (Attempt: 1/10)
For those of you playing at home, here's what we're about to run in order to convert the mailbox: /usr/bin/dsync -o mail_fsync=never -o maildir_broken_filename_sizes=yes -o mailbox_list_index=no -o mail_location=maildir:/home/domaintest/mail -u [email protected] -v mirror -x INBOX..* -x *@*_* mdbox:/home/domaintest/mail
Syncing while existing sessions for “[email protected]” are using mdbox (pass 2).
Syncing “[email protected]” mdbox -> maildir (Attempt: 1/10)
For those of you playing at home, here's what we're about to run in order to convert the mailbox: /usr/bin/dsync -o mail_fsync=never -o maildir_broken_filename_sizes=yes -o mailbox_list_index=no -o mail_location=maildir:/home/domaintest/mail -u [email protected] -v mirror -x INBOX..* -x *@*_* mdbox:/home/domaintest/mail
Removing conversion lock files for “[email protected]”
Flushing the authentication cache for dovecot.
Bouncing existing session for “[email protected]” in order to have them reload in the new format.
Syncing again to pickup any new mail that was delivered while the system converted “[email protected]” to maildir.
Syncing “[email protected]” mdbox -> maildir (Attempt: 1/10)
For those of you playing at home, here's what we're about to run in order to convert the mailbox: /usr/bin/dsync -o maildir_broken_filename_sizes=yes -o mailbox_list_index=no -o mail_location=maildir:/home/domaintest/mail -u [email protected] -v mirror -x INBOX..* -x *@*_* mdbox:/home/domaintest/mail
Removing mdbox files for “[email protected]” from /home/domaintest/mail.
rmdir /home/domaintest/mail/mailboxes
rmdir /home/domaintest/mail/storage
[email protected]: success
Changing directory to “/home/domaintest/mail/domaintest.com/acct”.
Target “/home/domaintest/mail/domaintest.com/acct” on host “cp001.sitesexclusivos.com.br” has 99.16 GB free and requires at least 67.94 KB free, which includes space for temporary files.
Creating conversion lock file for “[email protected]”
Cleaning up “/home/domaintest/mail/domaintest.com/acct” for “[email protected]”.
Syncing while existing sessions for “[email protected]” are using mdbox (pass 1).
Syncing “[email protected]” mdbox -> maildir (Attempt: 1/10)
For those of you playing at home, here's what we're about to run in order to convert the mailbox: /usr/bin/dsync -o mail_fsync=never -o maildir_broken_filename_sizes=yes -o mailbox_list_index=no -o mail_location=maildir:/home/domaintest/mail/domaintest.com/acct -u [email protected] -v mirror -x INBOX..* -x *@*_* mdbox:/home/domaintest/mail/domaintest.com/acct
Syncing while existing sessions for “[email protected]” are using mdbox (pass 2).
Syncing “[email protected]” mdbox -> maildir (Attempt: 1/10)
For those of you playing at home, here's what we're about to run in order to convert the mailbox: /usr/bin/dsync -o mail_fsync=never -o maildir_broken_filename_sizes=yes -o mailbox_list_index=no -o mail_location=maildir:/home/domaintest/mail/domaintest.com/acct -u [email protected] -v mirror -x INBOX..* -x *@*_* mdbox:/home/domaintest/mail/domaintest.com/acct
Removing conversion lock files for “[email protected]”
Flushing the authentication cache for dovecot.
Bouncing existing session for “[email protected]” in order to have them reload in the new format.
Syncing again to pickup any new mail that was delivered while the system converted “[email protected]” to maildir.
Syncing “[email protected]” mdbox -> maildir (Attempt: 1/10)
For those of you playing at home, here's what we're about to run in order to convert the mailbox: /usr/bin/dsync -o maildir_broken_filename_sizes=yes -o mailbox_list_index=no -o mail_location=maildir:/home/domaintest/mail/domaintest.com/acct -u [email protected] -v mirror -x INBOX..* -x *@*_* mdbox:/home/domaintest/mail/domaintest.com/acct
Removing mdbox files for “[email protected]” from /home/domaintest/mail/domaintest.com/acct.
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Trash/dbox-Mails/dovecot.index.log
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Trash/dbox-Mails/dovecot.index.cache
rmdir dbox-Mails
rmdir Trash
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/spam/dbox-Mails/dovecot.index.log
rmdir dbox-Mails
rmdir spam
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Archive/dbox-Mails/dovecot.index.log
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Archive/dbox-Mails/dovecot.index.cache
rmdir dbox-Mails
rmdir Archive
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Atenção/dbox-Mails/dovecot.index.log
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Atenção/dbox-Mails/dovecot.index.cache
rmdir dbox-Mails
rmdir Atenção
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Atencao/dbox-Mails/dovecot.index.log
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Atencao/dbox-Mails/dovecot.index.cache
rmdir dbox-Mails
rmdir Atencao
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Sent/dbox-Mails/dovecot.index.log
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Sent/dbox-Mails/dovecot.index.cache
rmdir dbox-Mails
rmdir Sent
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Drafts/dbox-Mails/dovecot.index.log
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Drafts/dbox-Mails/dovecot.index.cache
rmdir dbox-Mails
rmdir Drafts
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/INBOX/dbox-Mails/dovecot.index.log
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/INBOX/dbox-Mails/dovecot.index.cache
rmdir dbox-Mails
rmdir INBOX
rmdir /home/domaintest/mail/domaintest.com/acct/mailboxes
unlink /home/domaintest/mail/domaintest.com/acct/storage/dovecot.map.index.log
unlink /home/domaintest/mail/domaintest.com/acct/storage/m.1
rmdir /home/domaintest/mail/domaintest.com/acct/storage
[email protected]: success
Restoring quota restrictions for “domaintest”.
Success.

Notice there are no mentions whatsoever that some folders were skipped or could not be read. And unfortunately I trusted Cpanel too much on this one that I didn't click the KEEP OLD MAILBOX FILES checkbox.

 

[cPanelLauren]

Hello,


You're right, this is definitely an issue and while it is partially fixed in a later version of cPanel/WHM with the internal case ID - CPANEL-28981 there are a number of open cases being worked to address this issue.

The full resolution for the issue will be in the internal cases CPANEL-26708 and CPANEL-26762 which are currently unresolved.

We'll do our best to update here when the issues are addressed but you can also follow along in our changelogs.

You can check them here: Change Logs - Change Logs - cPanel Documentation

 

[Denis Gomes Franco]

That's great news. However in the meantime, while that bug isn't fixed, maybe someone could put up a simple warning before conversion (this warning might get skimped over, but then you guys did your part so...), or add a check before conversion, or make the KEEP OLD MAILBOX FILES option mandatory.

 

[cPanelLauren]

That's great news. However in the meantime, while that bug isn't fixed, maybe someone could put up a simple warning before conversion (this warning might get skimped over, but then you guys did your part so...), or add a check before conversion, or make the KEEP OLD MAILBOX FILES option mandatory.

A UI warning such as that is suggested in one of the cases but not implemented yet.

 

[cPanelLauren]

Hello,

This is just an update to let you know that this issue is marked as resolved in v86 of cPanel & WHM and can be referenced in our changelogs here: Change Logs - Change Logs - cPanel Documentation