In Progress [CPANEL-31965] Default email account ghost disk usage

[saad92]

Any update on this issue? I'm having the same problem.

 

[GlobaltechBR]

@cPanelLauren
We are having the same problems reported in these topics. Does version on WHM 90.0.5 correct these issues?

 

[Ahmed Saleh92]

Support Ticket ID is: 93687839
[email protected] [~]# cd /home/dotsmedia/mail/
[email protected] [mail]# du -h --max-depth=1
8.0K ./new
28K ./.Archive
88K ./cur
20G ./dotsmedia.agency
24K ./.Drafts
24K ./.spam
4.0K ./tmp
24K ./.Junk
24K ./.Sent
24K ./.Trash
20G .
@cPanelLauren

 

[cPanelLauren]

I checked in on this case and it is not resolved. The workaround being provided is listed in this thread but I've quoted it below:

Hey guys,

I hate to say it but it looks like this issue has sprouted a new case CPANEL-31965 in which users are experiencing this behavior once again in v86 - I must have missed it when it was opened so I wasn't aware of it making a resurgence. @benito thank you for opening the ticket, the case is currently marked as Needs Reproduction and I've reached out to the product owner of that team to see if he needs anything from your server to reproduce this. We'll let you know what they come back with.

Right now the only workaround for this is to run the following:

/scripts/generate_maildirsize --confirm $USER

But this is only temporary as if you access the default email account the issue will reoccur.

Our Senior Technical Analysts do still need reproduction on this and I've pointed them in the direction of your ticket. They should be getting back to you soon!

 

[4u123]

This issue is still happening on all our servers and is identical to the previous case that was marked as resolved. The default account may be using no space at all but often shows as several GB in size.

This not only happens on all our existing servers but also on new servers where accounts have been copied over from older servers. We though that perhaps the transfer process would "reset" this issue but sadly not.

I wonder if it is related to compression. If it helps, we're using a compression level of 7 rather than the default 6.

To add further comment for discussion, there does seem to be strange output to the user for the mailbox sizes. For example, in the Disk usage tool, the uncompressed totals are displayed. Why? The file size of a compressed file is what it is. If you upload a zip file, it's file size is calculated based on the size on disk -not the size it would be if it was unpacked. So why provide such confusing information to the end user? Makes no sense.

Below, the user has a 10GB quota and it shows they are using 16GB in their Email accounts. But they are not using that much space - the files are compressed - so they take up less space. What's the point in showing the file size of what the files would be if they were not compressed? It serves no purpose at all!

This is a classic example of developers being developers and not taking into account that the end user doesn't understand, care, or need to know about compression. All you are doing is presenting confusing figures that ultimately result in a poor experience for the end user and more support tickets for us to deal with.

Email Accounts†	16,499.26 MB	* Contained in the mail directory.
	18,398.74 MB total disk space used.
	10,240.00 MB quota limit (9,160.47 MB used).

 

[cPanelLauren]

From what I understand it appears that the issue is only affecting accounts added prior to the fix for the issue that was implemented in v82 - meaning that the fix is only resolving the issue on accounts added after the fix rather than resolving it for existing ones.

Running:

/usr/local/cpanel/scripts/generate_maildirsize --confirm $user

Has been proven to resolve the issue for the accounts in this situation.


I'm confirming right now with our senior analysts that this includes accounts that have been transferred. I'll let you know what I find out.

 

[4u123]

From what I understand it appears that the issue is only affecting accounts added prior to the fix for the issue that was implemented in v82 - meaning that the fix is only resolving the issue on accounts added after the fix rather than resolving it for existing ones.

So you are saying that the issue of the default account total being incorrect is only happening on accounts added before the v82 fix and this problem no longer happens on new accounts. Meaning that the fix was put in place to stop it happening going forward but it won't resolve any existing accounts with this problem.

Therefore, running /usr/local/cpanel/scripts/generate_maildirsize --confirm $user should resolve the incorrect display of totals on the old accounts and the v82 fix should prevent it from recurring?

However, many have reported running that script, only to find the totals go wonky again soon after. So what we need of course is a permanent fix for the issue on the pre v82 accounts.

Lauren, could you possibly get a comment from your higher ups about the decision to show the uncompressed sizes for display instead of the actual disk usage? It's pretty clear that this is causing a great deal of confusion for customers. There appears to be no benefit to anyone in doing this. I'd like to hear some justification for it if possible.

 

[cPanelLauren]

Right now, you've got it. Up until the last few weeks we actually couldn't replicate the issue, until it was found that it was only occurring on accounts that got added before v82. The case is still open, that's not the official fix it's a workaround until the issue is resolved. I agree that you need a permanent resolution. We're hoping now that we have a reliable method of replication that development will be able to identify a suitable solution.


As far as the compression issue, I can open an inquiry with development to find out why it was done this way, I don't disagree that it is confusing

 

[amet]

Hello all,

just to add to this, the only way I can get the email usage displayed correctly is to run "/usr/local/cpanel/scripts/generate_maildirsize --confirm $user" as mentioned above, but .cpanel/email_accounts.json" also needs to be removed to show the correct usage. all this has to be done daily to get correct usage displayed.

I am currently on v90.0.10 but al accounts have been made on v80 or so

 

[cPanelLauren]

@4u123

I did some digging on the compression discrepancy and found the following very detailed explanation:

This makes sense for the purpose of disk quotas, as filesystem blocks cannot be shared by multiple files, so all of this space must be accounted to the system user which owns the file, even if the programs associated with the data aren't actually using that leftover space. This is a form of so-called "internal" fragmentation. If blocks could be shared by multiple files, and if these files could be owned by multiple system users, this would change the way the system would calculate disk quotas.

On the other hand, Dovecot cannot make assumptions about the underlying storage mechanism. Under Maildir or mbox, Dovecot on Linux could in theory ask the system about the number of filesystem blocks actually in use by particular files; however, the Dovecot software is not specific to Linux, so it does not make sense to make assumptions that its storage will only be allocated in blocks of a particular size. Moreover, this would add extreme amounts of complexity for little benefit. Therefore, it is much easier and makes more sense for it to account for usage in terms of the actual size of messages, regardless of how they are stored. For this reason, it is exceedingly unlikely that the usage reported by the system will exactly match the usage reported by Dovecot.

That said, this is not the only source of difference between the two figures at work here. Compression does not affect the size of messages, which never changes, even though it does change the amount of storage Dovecot needs for a given message. Additionally, compression at the point of storage is not a native capability of Dovecot and is instead implemented as a plugin. Changing Dovecot to be able to account for compression when accounting for usage against quotas likely would require additional complexity without sufficient benefit.

Both of these figures are correct, because they measure two quantities which are entirely different but related. The use of compression wildly exacerbates the difference, but even without use of compression, differences could be significant due to internal fragmentation. It just so happens that cPanel chooses to use one figure in certain places and the other figure in others.

 

[4u123]

@4u123

I did some digging on the compression discrepancy and found the following very detailed explanation:

Wow - who knew it could be so complicated? Sadly that leaves the end user in a very confusing situation. Like I said before - they don't care about the different ways that nerds can calculate usage - they just want to know how much disk space they are using.

Surely for the end user, all that's required is to calculate the file usage the same way as everything else. Why can't you simply calculate the usage in the mail folders according to the quotas - why wouldn't that be enough?

To say "both totals are correct" is not right. This isn't a science convention, it's a commercial product. Both totals are not correct. There can only be one total! The message sizes are irrelevant because the messages are compressed.

The reason you display the totals in cpanel is solely to let the client know how much usage they are using. They don't need to know that dovecot provides a different total based on the actual message size as it would be when not compressed - they just don't need to know this info.

In my example above it says...

18,398.74 MB total disk space used.

No - that's not true. If that much space had actually been used, it couldn't be allocated for anything else and would therefore be the actual total. But it isn't.

10,240.00 MB quota limit (9,160.47 MB used).

9,160.47 MB is the total disk space used as far as the end user is concerned. That's how much space on disk the client is using. You don't need the other figure at all. Why even display it?

As I said before and didn't receive an answer... I would really like to hear some justification for this. It's clearly nuts.

 

[iamkenrocks]

Hello Cpanel,

Greetings from the beautiful island of the Philippines.

Any update on this, please. It's a brand new year:

It appears that the default email account usage is showing again as 6.23GB after running a script ( /scripts/generate_maildirsize --confirm --allaccounts --verbose [cpaneluser]

Please refer to the screenshot below.





Running a script to recalculate the usage for the default email will make it show 0 MB for a while. But still, eventually, the large number comes back for the empty default account. We dig further and found out that this is an ongoing concern based on Cpanel Forums. Refer to this link https://forums.cpanel.net/threads/c...account-showing-incorrect-usage.649293/page-3

The disk usage shown in this interface is not actually impacting the cPanel account's disk usage. The default email adds all email quota from all other email accounts the user have, this is the reason why it shows as 6.23 GB. The screenshot below shows the actual usage of the default email account.

Please advise why this is happening. Looking forward to hearing from you guys. More power

 

[cPRex]

@iamkenrocks - I just checked the case and I see our developers are still working on it, although there isn't a fix available just yet.

 

[iamkenrocks]

@iamkenrocks - I just checked the case and I see our developers are still working on it, although there isn't a fix available just yet.

Thank you @cPRex for the update and I will wait until I hear from you on the next. The case was first reported in 2019, and now it's 2021 I believe this will be sorted this year yeah? Thank you again cpanel and we wish you a happy life. Charot.

 

[cPRex]

@iamkenrocks - kind of :D We've had a few different versions of the same issue appear, and they all have required slightly different resolutions on the development side. The current version of the problem was identified in March 2020, as the other issues from 2019 have been resolved.

So there has been progress since then, but this new one still has yet to be fixed. I'm watching it though and I'll post as soon as I get more details!

 

[jp208834]

Has this ever been resolved. We are having this issue with a few accounts again.

 

[cPRex]

@jp208834 - we haven't had any reports of this issue for some time. If you're able to reproduce this on a server currently, could you please open a support ticket with our team so we can check the system?