Hey everyone. I think I found a rather unusual, and possibly quite rare, bug.
For quite some time I've been using the MDBOX format due to its efficiency. But I had to convert some accounts back to MAILDIR format.
And then I found out - the hard way, after a customer's complaint - that while the conversion from MAILDIR to MDBOX was flawless, the opposite has a weird bug: all mails stored in folders that had accented characters (eg. FINALIZAÇÃO) were lost.
Thing is, during conversion Cpanel didn't show any warnings. Those folders simply weren't converted and now they are gone. Below I'll summarize the steps to reproduce this issue:
1. Create a CPANEL account using the MDBOX format
2. Create an e-mail account
3. Open that e-mail account in Roundcube
4. Create two folders, one named FINALIZACAO (without accents) and another named FINALIZAÇÃO (with accents)
5. Save some messages to both these folders
6. Run the conversion from MDBOX to MAILDIR
7. Fire up Roundcube again
8. Notice there is only the folder named FINALIZACAO.
Here is the log of the conversion process I ran in order to test this bug:
Notice there are no mentions whatsoever that some folders were skipped or could not be read. And unfortunately I trusted Cpanel too much on this one that I didn't click the KEEP OLD MAILBOX FILES checkbox.
For quite some time I've been using the MDBOX format due to its efficiency. But I had to convert some accounts back to MAILDIR format.
And then I found out - the hard way, after a customer's complaint - that while the conversion from MAILDIR to MDBOX was flawless, the opposite has a weird bug: all mails stored in folders that had accented characters (eg. FINALIZAÇÃO) were lost.
Thing is, during conversion Cpanel didn't show any warnings. Those folders simply weren't converted and now they are gone. Below I'll summarize the steps to reproduce this issue:
1. Create a CPANEL account using the MDBOX format
2. Create an e-mail account
3. Open that e-mail account in Roundcube
4. Create two folders, one named FINALIZACAO (without accents) and another named FINALIZAÇÃO (with accents)
5. Save some messages to both these folders
6. Run the conversion from MDBOX to MAILDIR
7. Fire up Roundcube again
8. Notice there is only the folder named FINALIZACAO.
Here is the log of the conversion process I ran in order to test this bug:
Code:
Starting “RESTORE” for “Account” “domaintest”.
The original files will be removed after the conversion.
Temporarily lifting quota restrictions for “domaintest”.
Changing directory to “/home/domaintest/mail”.
Target “/home/domaintest/mail” on host “cp001.sitesexclusivos.com.br” has 99.16 GB free and requires at least 0 bytes free, which includes space for temporary files.
Creating conversion lock file for “[email protected]”
Cleaning up “/home/domaintest/mail” for “[email protected]”.
Syncing while existing sessions for “[email protected]” are using mdbox (pass 1).
Syncing “[email protected]” mdbox -> maildir (Attempt: 1/10)
For those of you playing at home, here's what we're about to run in order to convert the mailbox: /usr/bin/dsync -o mail_fsync=never -o maildir_broken_filename_sizes=yes -o mailbox_list_index=no -o mail_location=maildir:/home/domaintest/mail -u [email protected] -v mirror -x INBOX..* -x *@*_* mdbox:/home/domaintest/mail
Syncing while existing sessions for “[email protected]” are using mdbox (pass 2).
Syncing “[email protected]” mdbox -> maildir (Attempt: 1/10)
For those of you playing at home, here's what we're about to run in order to convert the mailbox: /usr/bin/dsync -o mail_fsync=never -o maildir_broken_filename_sizes=yes -o mailbox_list_index=no -o mail_location=maildir:/home/domaintest/mail -u [email protected] -v mirror -x INBOX..* -x *@*_* mdbox:/home/domaintest/mail
Removing conversion lock files for “[email protected]”
Flushing the authentication cache for dovecot.
Bouncing existing session for “[email protected]” in order to have them reload in the new format.
Syncing again to pickup any new mail that was delivered while the system converted “[email protected]” to maildir.
Syncing “[email protected]” mdbox -> maildir (Attempt: 1/10)
For those of you playing at home, here's what we're about to run in order to convert the mailbox: /usr/bin/dsync -o maildir_broken_filename_sizes=yes -o mailbox_list_index=no -o mail_location=maildir:/home/domaintest/mail -u [email protected] -v mirror -x INBOX..* -x *@*_* mdbox:/home/domaintest/mail
Removing mdbox files for “[email protected]” from /home/domaintest/mail.
rmdir /home/domaintest/mail/mailboxes
rmdir /home/domaintest/mail/storage
[email protected]: success
Changing directory to “/home/domaintest/mail/domaintest.com/acct”.
Target “/home/domaintest/mail/domaintest.com/acct” on host “cp001.sitesexclusivos.com.br” has 99.16 GB free and requires at least 67.94 KB free, which includes space for temporary files.
Creating conversion lock file for “[email protected]”
Cleaning up “/home/domaintest/mail/domaintest.com/acct” for “[email protected]”.
Syncing while existing sessions for “[email protected]” are using mdbox (pass 1).
Syncing “[email protected]” mdbox -> maildir (Attempt: 1/10)
For those of you playing at home, here's what we're about to run in order to convert the mailbox: /usr/bin/dsync -o mail_fsync=never -o maildir_broken_filename_sizes=yes -o mailbox_list_index=no -o mail_location=maildir:/home/domaintest/mail/domaintest.com/acct -u [email protected] -v mirror -x INBOX..* -x *@*_* mdbox:/home/domaintest/mail/domaintest.com/acct
Syncing while existing sessions for “[email protected]” are using mdbox (pass 2).
Syncing “[email protected]” mdbox -> maildir (Attempt: 1/10)
For those of you playing at home, here's what we're about to run in order to convert the mailbox: /usr/bin/dsync -o mail_fsync=never -o maildir_broken_filename_sizes=yes -o mailbox_list_index=no -o mail_location=maildir:/home/domaintest/mail/domaintest.com/acct -u [email protected] -v mirror -x INBOX..* -x *@*_* mdbox:/home/domaintest/mail/domaintest.com/acct
Removing conversion lock files for “[email protected]”
Flushing the authentication cache for dovecot.
Bouncing existing session for “[email protected]” in order to have them reload in the new format.
Syncing again to pickup any new mail that was delivered while the system converted “[email protected]” to maildir.
Syncing “[email protected]” mdbox -> maildir (Attempt: 1/10)
For those of you playing at home, here's what we're about to run in order to convert the mailbox: /usr/bin/dsync -o maildir_broken_filename_sizes=yes -o mailbox_list_index=no -o mail_location=maildir:/home/domaintest/mail/domaintest.com/acct -u [email protected] -v mirror -x INBOX..* -x *@*_* mdbox:/home/domaintest/mail/domaintest.com/acct
Removing mdbox files for “[email protected]” from /home/domaintest/mail/domaintest.com/acct.
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Trash/dbox-Mails/dovecot.index.log
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Trash/dbox-Mails/dovecot.index.cache
rmdir dbox-Mails
rmdir Trash
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/spam/dbox-Mails/dovecot.index.log
rmdir dbox-Mails
rmdir spam
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Archive/dbox-Mails/dovecot.index.log
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Archive/dbox-Mails/dovecot.index.cache
rmdir dbox-Mails
rmdir Archive
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Atenção/dbox-Mails/dovecot.index.log
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Atenção/dbox-Mails/dovecot.index.cache
rmdir dbox-Mails
rmdir Atenção
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Atencao/dbox-Mails/dovecot.index.log
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Atencao/dbox-Mails/dovecot.index.cache
rmdir dbox-Mails
rmdir Atencao
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Sent/dbox-Mails/dovecot.index.log
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Sent/dbox-Mails/dovecot.index.cache
rmdir dbox-Mails
rmdir Sent
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Drafts/dbox-Mails/dovecot.index.log
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/Drafts/dbox-Mails/dovecot.index.cache
rmdir dbox-Mails
rmdir Drafts
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/INBOX/dbox-Mails/dovecot.index.log
unlink /home/domaintest/mail/domaintest.com/acct/mailboxes/INBOX/dbox-Mails/dovecot.index.cache
rmdir dbox-Mails
rmdir INBOX
rmdir /home/domaintest/mail/domaintest.com/acct/mailboxes
unlink /home/domaintest/mail/domaintest.com/acct/storage/dovecot.map.index.log
unlink /home/domaintest/mail/domaintest.com/acct/storage/m.1
rmdir /home/domaintest/mail/domaintest.com/acct/storage
[email protected]: success
Restoring quota restrictions for “domaintest”.
Success.