Mailbox Compression and Maildir-->MDBox conversion

[mtindor]

Hello,

I just enabled Mailbox compression today. 99% of the mailboxes on said server are using Maildir. I want to gradually migrate, per account, users' email from the maildir format to the mdbox format.

New emails coming in are automatically being compressed. Old emails are not automatically compressed (and yes, I realized this is documented).

I wanted to convert an account from maildir to mdbox and have all messages compressed. So I ran the maildir-to-mdbox conversion, then ran an mdbox-to-maildir conversion, and then ran a maildir-to-mdbox conversion.

No matter what I do, the conversion process does not seem to be compressing all of the old email for an account. But, from what I've read of others on the forum, simply converting from maildir to mdbox (or vice versa) and back should cause all of the email for a specific account to be compressed.

What am I doing wrong?

Mike

 

[cPanelMichael]

Hi Mike,

It's important to note that we don't offer support for the compression of existing messages at this time. The feature request for this is open at:

Ability to compress existing email

I encourage you to vote and add your feedback to this request. As far as a temporary workaround to compress existing messages, some users have reported success with this by switching from the Maildir format to the Mdbox format, back to the Maildir format, and then to the Mdbox format again. However, it's important to note the following warning from our Mailbox Conversion document if you choose to do this:

When you convert from one mailbox format to another, the unique identifier (UID) for each email message on the server will change. The system uses UIDs to identify each message that a POP email account receives. Email clients that do not delete mail from the server use these UIDs to determine which messages to download. Due to this change in UIDs, mailbox conversions can cause POP email users to receive previously-downloaded messages as though they were new.

If the temporary workaround isn't working for you, could you verify how you are checking the results?

Thank you.

 

[mtindor]

I recognize that cPanel doesn't support the ability to compress existing email at this time. I have followed exactly that process which others have outlined in the forums. Yes I'm aware of message indexes getting reset for POP3 users.

I verified that it is not actually compressing all mail when converting back to maildir and then converting to mdbox again by attempting to view the various m.* files in the storage folder. I can read all of the messages in all of the m.* files using cat / less without issue. In the last m.* file (in this case m.91), there is a mixture of uncompressed and compressed messages because that particular file contains messages from prior to this morning + a couple from this morning [after I enabled compression].

If the conversion back to maildir and then forward to mdbox were actually compressing all of the email, I would not be able to view all of the messages in any of the m.* files using cat / less.

Mike

 

[cPanelMichael]

I verified that it is not actually compressing all mail when converting back to maildir and then converting to mdbox again by attempting to view the various m.* files in the storage folder. I can read all of the messages in all of the m.* files using cat / less without issue. In the last m.* file (in this case m.91), there is a mixture of uncompressed and compressed messages because that particular file contains messages from prior to this morning + a couple from this morning [after I enabled compression].

If the conversion back to maildir and then forward to mdbox were actually compressing all of the email, I would not be able to view all of the messages in any of the m.* files using cat / less.

Could you open a support ticket using the link in my signature so we can take a closer look at the system?

Thank you.

 

[mtindor]

I did upvote the feature request. Thanks for showing me that. Of course, unless a lot of people upvote it, it'll be useless.

At any rate, after further searching, I see that even now after dovecot has supported compression for years there is still no simple, tried-and-true reliable way of compressing existing mail.

I have to wonder if those who report successful compression of all emails have really checked and confirmed that every indidivual email file (maildir) or m.* file (mdbox) is indeed compressed. Perhaps the space savings they are seeing was simply a savings due to conversion from maildir to mdbox. Of course, I saw one person mention going from 5.5 GB to 3.5 GB of space used. I can't imagine that one could realize that kind of space savings in any other way but having full compressed mail files. I can't imagine saving that amount of space purely by changing from maildir to mdbox (without compression of existing email).

Mike

 

[mtindor]

Could you open a support ticket using the link in my signature so we can take a closer look at the system?

Thank you.

Hi Michael,

I've opened a support ticket.

Thanks

Mike

 

[mtindor]

As I suspected, cPanel support has verified that switching back and forth between the maildir / mdbox formats is not going to magically make all of your mail become compressed, even if you are sure compression is enabled in dovecot, since the conversion scripts do not pass anything through Dovecot and that is the only way Dovecot would compress them.

Mike

 

[Hedloff]

Hi Mike,

It's important to note that we don't offer support for the compression of existing messages at this time. The feature request for this is open at:

Ability to compress existing email

I cannot seem to find this feature request, was it removed? Why?

Did you find anyway to compress old emails Mike?

 

[cPanelMichael]

Hello @Hedloff,

Thanks for the report. It looks to have been removed as part of the feature request website's hardware failure in March:

On March 28th, 2018 at approximately 1:30 pm CST the server housing the cPanel Feature Request site’s hypervisor unexpectedly failed. This failure resulted in the loss of data back to September of 2017. Read more in our announcement.

I've re-created the feature request, and it should be approved out of moderation shortly. Once it's approved, please take the time to vote and re-add any feedback to the request:

https://features.cpanel.net/topic/ability-to-compress-existing-email

We apologize for the inconvenience.

Thank you.

 

[petersphilo]

Sorry for reviving an old thread..
Does anyone have an idea of how much space can be saved, and if users connected using IMAP will also be forced to re-download all emails? (note that docs explicitly state that POP users will re-download all emails present on the server because of new UUIDs on each message)

i have clients with mailboxes from 20+GB to 70+GB -- before you ask, i have no idea what in the world they keep in there, but i'm pretty sure it's not giant videos; they're more like pack-rats, keeping every email they ever got, all those email with small files attached add up over time...

How much space can i reasonably expect to save?
is it 1%; 8%; 25%?

i already have compression level set to 6 in 'Mailserver Configuration'...

Also, i can't ask my clients to re-download their entire 70GB worth of emails; especially if all of it is going to show up as unread...
(On that note, what about custom folders; will their contents also be re-downloaded?)

Any help would be greatly appreciated!!
Thank you!

 

[cPanelMichael]

Hello @petersphilo,

I've merged your post into the existing thread on this topic. You can review the previous posts to this thread for discussion related to your questions.

Thank you.

 

[VIRTBIZ CHRIS]

I just stumbled upon this old thread and have to say I'm surprised there hasn't been more traction. Email consumes a pretty large amount of space on our systems, and the ability to compress existing stored mail would be very welcome.